Architecture Overview
Deploying this solution builds the following environment in the AWS Cloud.
Figure 1: Cross-Region Replication Monitor architecture on AWS
crr-monitor Template
The crr-monitor.template is the solution's primary AWS CloudFormation template. This
template deploys the components to enable monitoring of cross-region replication either
within a single AWS account or across AWS accounts. Monitoring across AWS accounts is
possible when the Amazon S3 source buckets reside in the
account where this template is deployed. The template deploys AWS Identity and Access Management (IAM) roles,
AWS Lambda functions, an AWS CloudTrail trail, an Amazon CloudWatch rule, Amazon Simple Notification Service (Amazon SNS) topics, an Amazon SQS
queue, and Amazon DynamoDB tables.
This solution enables AWS CloudTrail and begins logging S3 events for the source and destination buckets where cross-region replication is configured. If new buckets are configured with cross-region replication after the solution has been deployed, they can be added quickly to CloudTrail using the AWS Management Console.
When an object is added to the S3 source bucket, AWS CloudTrail logs the event data. This activity triggers a CloudWatch event rule that publishes a message to the Amazon SNS topic in the source bucket’s region. An Amazon SQS queue subscribed to the Amazon SNS topic receives the message for processing. Once the object replication to the destination bucket is successful, status information is sent back to the Amazon SQS queue.
A CloudWatch event automatically invokes the CRRMonitor Lambda function. By default, the function is invoked every 60 seconds but this is configurable in the CloudWatch console. The function reads messages from the SQS queue and updates the CRRMonitor DynamoDB table with the replication status. Status data in the DynamoDB table is deleted and replaced every 24 hours.
crr-agent Template
Deploy the crr-agent template in AWS accounts containing S3 source bucket(s) when you have the crr-monitor
template deployed in a separate AWS account.
The crr-monitor template is deployed in the AWS account containing the
destination bucket(s).
The crr-agent template deploys a CloudWatch rule that forwards events to
the AWS account event bus (where the crr-monitor template is deployed, or the Monitor account) to track the replication status. After the event
is delivered to the Monitor account, the operations
described in the crr-monitor Template section are performed. If the AWS account containing
the S3 source buckets does not have CloudTrail
monitoring, you can configure AWS CloudTrail data events to monitor them.
Figure 2: Cross-Region Replication Agent architecture on AWS
When an object is added to the Amazon S3 source bucket, AWS CloudTrail logs the data event. This activity triggers an Amazon CloudWatch event rule that delivers the status information to the CloudWatch Logs in the Monitor account using an event bus. In the Monitor account, CloudWatch Logs sends the event to Amazon SNS. An Amazon SQS queue subscribed to the Amazon SNS topic receives the message for processing. After the object replication to the destination bucket is successful, the replication triggers a similar event, and sends the status information back to the Amazon SQS queue in the Monitor account.
After the AWS Lambda function verifies an object was successfully replicated, it stores the data in an Amazon DynamoDB table for immediate access. Status data in the DynamoDB table is deleted and replaced every 24 hours.
Customers who deploy this solution in an AWS Region that offers Amazon Kinesis Data Firehose can choose to archive solution data to Amazon S3. If you enable this feature, the solution uses a Firehose delivery stream to upload data to one of your existing S3 buckets for later analysis. You can use Amazon Athena
