Copy data from an S3 bucket in one account and Region to another account and Region - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsRelated resourcesAttachments

Copy data from an S3 bucket in one account and Region to another account and Region

Created by Appasaheb Bagali (AWS)

Environment: Production

Technologies: Cloud-native; Security, identity, compliance; Storage & backup; Migration

Workload: All other workloads

AWS services: Amazon S3

Summary

This pattern describes how to copy data from an Amazon Simple Storage Service (Amazon S3) bucket in an Amazon Web Services (AWS) account and AWS Region to an S3 bucket in another account and Region.

This pattern uses a source account and a destination account in different Regions. You attach a bucket policy to your source S3 bucket that grants the destination account access through AWS Identity and Access Management (IAM). You then create an IAM policy in your destination account that allows a user to perform PutObject and GetObject actions on the source S3 bucket. Finally, you run copy and sync commands to transfer data from the source S3 bucket to the destination S3 bucket.

Accounts own the objects that they upload to S3 buckets. If you copy objects across different accounts and Regions, you grant the destination account ownership of the copied objects. You can change the ownership of an object by changing its access control list (ACL) to bucket-owner-full-control. However, we recommend that you grant programmatic cross-account permissions to the destination account because ACLs can be difficult to manage for multiple objects.

Prerequisites and limitations

Prerequisites 

  • Two active AWS accounts in different AWS Regions.

  • An existing S3 bucket in the source account. 

Architecture

Technology stack

  • Amazon S3

  • IAM

  • AWS CLI 

Tools

  • Amazon S3 – Amazon Simple Storage Service (Amazon S3) is storage for the internet. 

  • AWS CLI – AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services.

  • AWS Identity and Access Management (IAM) – IAM is a web service that helps you securely control access to AWS resources.

Code 

Copy

aws s3 cp s3:// DOC-EXAMPLE-BUCKET-SOURCE / \
       s3:// DOC-EXAMPLE-BUCKET-TARGET / \
      --recursive --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

Synchronize

 aws s3 sync  s3:// DOC-EXAMPLE-BUCKET-SOURCE / \
       s3:// DOC-EXAMPLE-BUCKET-TARGET / \
     --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

Epics

Task Description Skills required
Create and attach an S3 bucket policy.

Sign in to the AWS Management Console for your source account and open the Amazon S3 console. Choose your source S3 bucket and then choose Permissions. Under Bucket policy, choose Edit and then paste the bucket policy from the sourcebucket-policy.json file (attached). Choose Save.

Important: Make sure that you include the AWS account ID for the destination account and configure the bucket policy template according to your requirements.

Cloud administrator
Task Description Skills required
Create a destination S3 bucket.

Sign in to the AWS Management Console for your destination account, open the Amazon S3 console, and then choose Create bucket. Create an S3 bucket according to your requirements. For help with this story, see Creating a bucket in the Amazon S3 documentation. 

Cloud administrator
Create an IAM policy for the IAM user.

Open the IAM console, choose Policies, and then choose Create policy. Choose the JSON tab and paste the destination-IAM-user-policy.json file (attached). Choose Review policy, enter a policy name and description, and then choose Create policy.

Choose either Programmatic access or AWS Management Console access. We recommend that you choose Programmatic access if users require access to the API, AWS CLI, or tools for Windows PowerShell. This creates an access key for each new user. You can view or download the access keys when you are on the confirmation page. 

Choose Next: Permissions and then choose Attach existing policies directly. Attach the policy that you created earlier. Choose Next: Tags, and then choose Next: Review to finalize the user configuration.

Cloud administrator
Task Description Skills required
Copy and synchronize data from the source S3 bucket to the destination S3 bucket.

Open AWS CLI and run the copy command from the Code section to copy the data from the source S3 bucket. 

Run the synchronize command from the Code section to transfer the data into your destination S3 bucket.

Your data is then copied from the source S3 bucket to the destination S3 bucket.

 Cloud administrator

Attachments

attachment.zip