Architecture Overview - Cross-Region Replication Monitor

Architecture Overview

Deploying this solution builds the following environment in the AWS Cloud.


        CRR Monitor Architecture on AWS

Figure 1: Cross-Region Replication Monitor architecture on AWS

crr-monitor Template

The crr-monitor.template is the solution's primary AWS CloudFormation template. This template deploys the components to enable monitoring of cross-region replication either within a single AWS account or across AWS accounts. Monitoring across AWS accounts is possible when the Amazon S3 source buckets reside in the account where this template is deployed. The template deploys AWS Identity and Access Management (IAM) roles, AWS Lambda functions, an AWS CloudTrail trail, an Amazon CloudWatch rule, Amazon Simple Notification Service (Amazon SNS) topics, an Amazon SQS queue, and Amazon DynamoDB tables.

This solution enables AWS CloudTrail and begins logging S3 events for the source and destination buckets where cross-region replication is configured. If new buckets are configured with cross-region replication after the solution has been deployed, they can be added quickly to CloudTrail using the AWS Management Console.

When an object is added to the S3 source bucket, AWS CloudTrail logs the event data. This activity triggers a CloudWatch event rule that publishes a message to the Amazon SNS topic in the source bucket’s region. An Amazon SQS queue subscribed to the Amazon SNS topic receives the message for processing. Once the object replication to the destination bucket is successful, status information is sent back to the Amazon SQS queue.

A CloudWatch event automatically invokes the CRRMonitor Lambda function. By default, the function is invoked every 60 seconds but this is configurable in the CloudWatch console. The function reads messages from the SQS queue and updates the CRRMonitor DynamoDB table with the replication status. Status data in the DynamoDB table is deleted and replaced every 24 hours.

crr-agent Template

Deploy the crr-agent template in AWS accounts containing S3 source bucket(s) when you have the crr-monitor template deployed in a separate AWS account.

Note

The crr-monitor template is deployed in the AWS account containing the destination bucket(s).

The crr-agent template deploys a CloudWatch rule that forwards events to the AWS account event bus (where the crr-monitor template is deployed, or the Monitor account) to track the replication status. After the event is delivered to the Monitor account, the operations described in the crr-monitor Template section are performed. If the AWS account containing the S3 source buckets does not have CloudTrail monitoring, you can configure AWS CloudTrail data events to monitor them.


          CRR Agent Architecture on AWS

Figure 2: Cross-Region Replication Agent architecture on AWS

When an object is added to the Amazon S3 source bucket, AWS CloudTrail logs the data event. This activity triggers an Amazon CloudWatch event rule that delivers the status information to the CloudWatch Logs in the Monitor account using an event bus. In the Monitor account, CloudWatch Logs sends the event to Amazon SNS. An Amazon SQS queue subscribed to the Amazon SNS topic receives the message for processing. After the object replication to the destination bucket is successful, the replication triggers a similar event, and sends the status information back to the Amazon SQS queue in the Monitor account.

After the AWS Lambda function verifies an object was successfully replicated, it stores the data in an Amazon DynamoDB table for immediate access. Status data in the DynamoDB table is deleted and replaced every 24 hours.

Note

Customers who deploy this solution in an AWS Region that offers Amazon Kinesis Data Firehose can choose to archive solution data to Amazon S3. If you enable this feature, the solution uses a Firehose delivery stream to upload data to one of your existing S3 buckets for later analysis. You can use Amazon Athena, a serverless, interactive query service, to easily analyze historical data in Amazon S3.