Copy data from an S3 bucket to another account and Region by using S3 Batch Replication - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resources

Copy data from an S3 bucket to another account and Region by using S3 Batch Replication

Created by Appasaheb Bagali (AWS), Lakshmikanth B D (AWS), Purushotham G K (AWS), Shubham Harsora (AWS), and Suman Rajotia (AWS)

Environment: PoC or pilot

Technologies: Storage & backup

AWS services: Amazon S3; AWS Identity and Access Management

Summary

This pattern explains how you can use Amazon Simple Storage Service (Amazon S3) Batch Replication to copy the contents of an S3 bucket to another S3 bucket automatically, without any manual intervention, after you set up the buckets. The source and destination buckets can be in the same or in different AWS accounts or Regions.

S3 Batch Replication gives you a way to replicate Amazon S3 objects that existed before a replication configuration was in place, objects that were previously replicated, and objects that failed replication. This method uses an S3 Batch Operations job. When the job finishes, you receive a completion report.

You can use S3 Batch Replication in scenarios that require continuous and automatic migration of new objects from a source bucket to a destination bucket. For one-time migration, you can use the AWS Command Line Interface (AWS CLI) instead, as described in the pattern Copy data from an S3 bucket to another account and Region by using the AWS CLI.

Prerequisites and limitations

  • A source AWS account.

  • A destination AWS account.

  • An S3 bucket in the source account with a few objects (files or folders).

  • One or more S3 buckets in the destination account.

  • S3 Versioning enabled on the source and destination buckets.

  • AWS Identity and Access Management (IAM) permissions to create an IAM policy, IAM role, and S3 bucket policy on the source and destination accounts.

  • Amazon S3 Lifecycle rules disabled while the S3 Batch Replication job is active. This ensures parity between the source and destination buckets. Otherwise, the destination bucket might not be an exact replica of the source bucket.

Architecture

Copying Amazon S3 objects to other accounts and Regions by using S3 Batch Replication

Tools

AWS services

Best practices

The following video from AWS re:Invent 2022 discusses best practices for using Amazon S3 replication for regulatory compliance, data protection, and increased application performance.

Epics

TaskDescriptionSkills required

Create an IAM policy for cross-account replication.

In the AWS source account:

  1. Open the IAM console.

  2. Create a new IAM policy.

  3. In the Policy editor section, choose JSON, and paste the following code.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GetSourceBucketConfiguration",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetBucketAcl",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::source-bucket-name",
                "arn:aws:s3:::source-bucket-name/*"
            ]
        },
        {
            "Sid": "ReplicateToDestinationBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:*Object",
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags"
            ],
            "Resource": [
                "arn:aws:s3:::destination-bucket-name*",
                "arn:aws:s3:::destination-bucket-name/*"
            ]
        },
        {
            "Sid": "PermissionToOverrideBucketOwner",
            "Effect": "Allow",
            "Action": [
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::destination-bucket-name*",
                "arn:aws:s3:::destination-bucket-name/*"
            ]
        }
    ]
}

    This policy includes three statements:

    • GetSourceBucketConfiguration provides access to the replication configuration and object version for replication on the source bucket.

    •  ReplicateToDestinationBuckets provides access to replicate to the destination bucket. You can specify multiple destination buckets in the array.

    • PermissionToOverrideBucketOwner provides access to ObjectOwnerOverrideToBucketOwner so that the destination bucket can own the objects in the destination account that were replicated from the source account.

  4. Choose Next, provide a policy name such as cross-account-bucket-replication-policy, and then choose Create policy.

For more information, see Creating IAM policies in the IAM documentation.

Cloud administrator, AWS administrator

Create an IAM role for cross-account replication.

In the AWS source account:

  1. On the IAM console, create an IAM role with the following information:

    1. For Trusted entity type, choose AWS service.

    2. For service, choose S3.

    3. For use case, choose S3 Batch Operations.

    4. Choose the policy you created in the previous step.

  2. Provide a role name such as cross-account-bucket-replication-role, and then choose Create role.

For more information, see Creating IAM roles in the IAM documentation.

Cloud administrator, AWS administrator
TaskDescriptionSkills required

Create a replication rule against the source bucket in the source account.

In the AWS source account:

  1. Open the Amazon S3 console.

  2. Navigate to the source bucket, and choose the Management tab.

  3. Create a replication rule with the following configuration:

    1. Provide a rule name such as s3-replication-rule.

    2. For Status, choose Enabled.

    3. For rule scope, choose Applies to all objects in the bucket.

    4. For Destination, choose Specify a bucket in another account, and then enter the destination AWS account number and the bucket name.

    5. Choose the option to change object ownership to the destination bucket owner.

    6. For IAM role, choose the role you created earlier in the source account.

    7. For Additional replication options, select all available options. These provide the ability to replicate content quickly, monitor the progress of replication through Amazon CloudWatch metrics, replicate delete markers, and replicate metadata changes.

    8. Choose Save.

  4. If you have multiple destination buckets, create additional replication rules.

For more information, see Configuring replication when source and destination buckets are owned by different accounts in the Amazon S3 documentation.

AWS administrator, Cloud administrator
TaskDescriptionSkills required

Apply a bucket policy to the destination bucket.

This step has to be performed for each destination bucket individually in AWS destination accounts.

In the AWS destination account:

  1. Open the IAM console, navigate to the destination bucket, and choose the Permissions tab.

  2. Edit the bucket policy by providing the following JSON code, and save the policy:

{
    "Version": "2012-10-17",
    "Id": "PolicyForDestinationBucket",
    "Statement": [
        {
            "Sid": "Permissions on objects and buckets",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::SourceAWSAccountNumber:role/IAM-Role-created-in-step1-in-source-account"
            },
            "Action": [
                "s3:List*",
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning",
                "s3:ReplicateDelete",
                "s3:ReplicateObject"
            ],
            "Resource": [
                "arn:aws:s3:::destination-bucket",
                "arn:aws:s3:::destination-bucket/*"
            ]
        },
        {
            "Sid": "Permission to override bucket owner",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::SourceAWSAccountNumber:role/IAM-Role-created-in-step1-in-source-account"
            },
            "Action": "s3:ObjectOwnerOverrideToBucketOwner",
            "Resource": "arn:aws:s3:::destination-bucket/*"
        }
    ]
}

This policy includes two statements:

  • Permissions on objects and buckets indicates that the destination bucket can replicate content based on the role defined in the source account. The role provides permissions to the source bucket.

  • Permission to override bucket owner indicates that the destination bucket has permissions to override the ownership from the source account.

AWS administrator, AWS systems administrator, Cloud administrator
TaskDescriptionSkills required

Verify that replication works correctly.

  1. Add an object to the source bucket.

  2. Verify that the new object appears in the S3 buckets in the destination accounts.

  3. View CloudWatch metrics:

    1. In the source bucket, choose the Metrics tab.

    2. In the Replication metrics section, select a replication rule.

    3. Choose Display charts. The charts reflect the state of replication by displaying the  operations that are pending replication, the replication latency, and the bytes pending replication.

For more information, see Monitoring metrics with Amazon CloudWatch in the Amazon S3 documentation.

AWS administrator, Cloud administrator

Related resources