Copy data from an S3 bucket to another account and Region by using the AWS CLI - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Copy data from an S3 bucket to another account and Region by using the AWS CLI

Created by Appasaheb Bagali (AWS) and Purushotham G K (AWS)

Environment: Production

Technologies: Storage & backup; CloudNative

AWS services: AWS CLI; AWS Identity and Access Management; Amazon S3

Summary

This pattern describes how to migrate data from an Amazon Simple Storage Service (Amazon S3) bucket in an AWS source account to a destination S3 bucket in another AWS account, either in the same AWS Region or in a different Region.

The source S3 bucket allows AWS Identity and Access Management (IAM) access by using an attached resource policy. A user in the destination account has to assume a role that has PutObject and GetObject permissions for  the source bucket. Finally, you run copy and sync commands to transfer data from the source S3 bucket to the destination S3 bucket.

Accounts own the objects that they upload to S3 buckets. If you copy objects across accounts and Regions, you grant the destination account ownership of the copied objects. You can change the ownership of an object by changing its access control list (ACL) to bucket-owner-full-control. However, we recommend that you grant programmatic cross-account permissions to the destination account because ACLs can be difficult to manage for multiple objects.

Warning: This scenario requires IAM users with programmatic access and long-term credentials, which present a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. Access keys can be updated if necessary. For more information, see Updating access keys in the IAM User Guide.

This pattern covers one-time migration. For scenarios that require continuous and automatic migration of new objects from a source bucket to a destination bucket, you can use S3 Batch Replication instead, as described in the pattern Copy data from an S3 bucket to another account and Region by using S3 Batch Replication.

Prerequisites and limitations

  • Two active AWS accounts in the same or different AWS Regions.

  • An existing S3 bucket in the source account. 

  • If your source or destination Amazon S3 bucket has default encryption enabled, you must modify the AWS Key Management Service (AWS KMS) key permissions. For more information, see the AWS re:Post article on this topic.

  • Familiarity with cross-account permissions.

Architecture

Copying Amazon S3 data to another account or Region

Tools

Best practices

Epics

TaskDescriptionSkills required

Create an IAM user and get the access key.

  1. Sign in to the AWS Management Console and create an IAM user that has programmatic access. For detailed steps, see Creating IAM users in the IAM documentation. There is no need to attach any policies for this user.

  2. Generate an access key and secret key for this user. For instructions, see AWS Account and Access Keys in the AWS documentation.

AWS DevOps

Create an IAM identity-based policy.

Create an IAM Identity-based policy named S3MigrationPolicy by using the following permissions. For detailed steps, see Creating IAM policies in the IAM documentation. 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectTagging",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket",
                "arn:aws:s3:::awsexamplesourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::awsexampledestinationbucket",
                "arn:aws:s3:::awsexampledestinationbucket/*"
            ]
        }
    ]
}

Note: Modify the source and destination bucket names according to your use case.

This identity-based policy allows the user who is assuming this role to access the source bucket and destination bucket.

AWS DevOps

Create an IAM role.

Create an IAM role named S3MigrationRole by using the following trust policy, and then attach the previously created S3MigrationPolicy. For detailed steps, see Creating a role to delegate permissions to an IAM user in the IAM documentation. 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<destination_account>:user/<user_name>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

Note: Modify the Amazon Resource Name (ARN) of the destination IAM role or user name in the trust policy according to your use case.

This trust policy allows the newly created IAM user to assume S3MigrationRole.

AWS DevOps
TaskDescriptionSkills required

Create and attach an S3 bucket policy.

Sign in to the AWS Management Console for your source account and open the Amazon S3 console. Choose your source S3 bucket and then choose Permissions. Under Bucket policy, choose Edit and then paste the following bucket policy. Choose Save.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::<destination_account>:role/<RoleName>"},
            "Action": ["s3:ListBucket",
                       "s3:GetObject",
                       "s3:GetObjectTagging",
                       "s3:GetObjectVersion",
                       "s3:GetObjectVersionTagging"
                       ],
            "Resource": [
                "arn:aws:s3:::awsexamplesourcebucket/*",
                "arn:aws:s3:::awsexamplesourcebucket"
            ]
        }
    ]
}

Note: Make sure that you include the AWS account ID for the destination account and configure the bucket policy template according to your requirements.

This resource-based policy allows the destination role S3MigrationRole to access S3 objects in the source account.

Cloud administrator
TaskDescriptionSkills required

Create a destination S3 bucket.

Sign in to the AWS Management Console for your destination account, open the Amazon S3 console, and then choose Create bucket. Create an S3 bucket according to your requirements. For more information, see Creating a bucket in the Amazon S3 documentation. 

Cloud administrator
TaskDescriptionSkills required

Configure AWS CLI with the newly created user credentials.

  1. Install the latest release of the AWS CLI. For instructions, see Installing or updating the latest version of the AWS CLI in the AWS CLI documentation.

  2. Run $ aws configure and update CLI with the AWS access key of the user you created. For more information, see Configuration and credential file settings in the AWS CLI documentation.

AWS DevOps

Assume the S3 migration role.

  1. Use the AWS CLI to assume S3MigrationRole:

     aws sts assume-role \
    --role-arn "arn:aws:iam::<destination_account>:role/S3MigrationRole" \
    --role-session-name AWSCLI-Session

    This command outputs several pieces of information. Inside the credentials block you need the AccessKeyId, SecretAccessKey, and SessionToken. This example uses the environment variables RoleAccessKeyID, RoleSecretKey, and RoleSessionToken. Note that the timestamp of the expiration field is in the UTC time zone. The timestamp indicates when the temporary credentials of the IAM role expire. If the temporary credentials expire, you must call the sts:AssumeRole API again.

  2. Create three environment variables to assume the IAM role. These environment variables are filled out with the following output:

    # Linux
export AWS_ACCESS_KEY_ID=RoleAccessKeyID
export AWS_SECRET_ACCESS_KEY=RoleSecretKey
export AWS_SESSION_TOKEN=RoleSessionToken
# Windows
set AWS_ACCESS_KEY_ID=RoleAccessKeyID
set AWS_SECRET_ACCESS_KEY=RoleSecretKey
set AWS_SESSION_TOKEN=RoleSessionToken

  3. Verify that you assumed the IAM role by running the following command:

    aws sts get-caller-identity

For more information, see the AWS Knowledge Center.

AWS administrator

Copy and synchronize data from the source S3 bucket to the destination S3 bucket.

When you have assumed the role S3MigrationRole you can copy the data using the copy (cp) or synchronize (sync) command.

Copy (see the AWS CLI Command Reference for details):

aws s3 cp s3:// DOC-EXAMPLE-BUCKET-SOURCE / \
    s3:// DOC-EXAMPLE-BUCKET-TARGET / \
    --recursive --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

Synchronize (see the AWS CLI Command Reference for details):

aws s3 sync s3:// DOC-EXAMPLE-BUCKET-SOURCE / \
    s3:// DOC-EXAMPLE-BUCKET-TARGET / \
    --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME
Cloud administrator

Troubleshooting

IssueSolution

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

  • Make sure you that you have assumed the role S3MigrationRole.

  • Run aws sts get-caller-identity to check the role used. If the output doesn’t display the ARN for S3MigrationRole, assume the role again and retry.

Related resources