Architecture overview - Amazon Marketing Cloud Uploader from AWS

Architecture overview

This section provides a reference implementation architecture diagram for the components deployed with this solution.

Architecture diagram

Deploying this solution with the default parameters deploys the following components in your AWS account.

Amazon Marketing Cloud Uploader from AWS architecture diagram

amc uploader diagram

The high-level process flow for the solution components deployed with the AWS CloudFormation template is as follows:

  1. User uploads first-party data to designated Amazon Simple Storage Service (Amazon S3) bucket or exports data from AWS Clean Rooms to the S3 bucket. Optionally, the user can designate data from the S3 bucket to an AWS Key Management Service (AWS KMS) key to decrypt and encrypt source data and its derivatives throughout the extract, transform, load (ETL) pipeline.

  2. User logs in with Amazon Cognito to the provided web application and obtains the authorization tokens needed to load frontend assets from Amazon S3 and backend resources from Amazon API Gateway.

  3. Users interact with the provided web application through an Amazon CloudFront distribution and an API Gateway endpoint. The CloudFront resource serves static website assets from Amazon S3. The API Gateway resource provides a REST API interface to the API handler AWS Lambda resource. This resource includes a variety of functions for creating, reading, updating, and deleting datasets. When an AWS KMS key is used to encrypt first-party data, this resource will also use the designated AWS KMS key to decrypt and read that data.

  4. The Amazon DynamoDB resource stores system configurations, such as user-specified connection details for Amazon Marketing Cloud instances. These configurations are inputs in the frontend web form and saved by the API handler Lambda resource. User-specified OAuth credentials are saved to AWS Secrets Manager as well as the programmatically-derived OAuth refresh token.

  5. The API handler Lambda resource interacts with one or more Amazon Marketing Cloud instances in order to create, read, update, and delete datasets.

  6. When users submit requests to upload data to new or existing datasets, the API handler Lambda resource starts an AWS Glue ETL job to normalize, hash, and reformat user-specified files according to the data upload rules of Amazon Marketing Cloud. The AWS Glue job will use the optionally designated AWS KMS key to decrypt the first-party data and encrypt transformed data objects when they are written to Amazon S3.

  7. The AWS Glue job outputs results to an ETL artifacts Amazon S3 bucket. This event initiates a request from the Uploader Lambda resource to each user-specified Amazon Marketing Cloud instance to initiate uploads of those results.

  8. Each user-specified Amazon Marketing Cloud instance asynchronously uploads transformed data objects from the ETL artifacts Amazon S3 bucket and uses the optionally designated AWS KMS key to decrypt those objects when needed.