/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EC2Permissions",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:CreateKeyPair",
"ec2:DescribeKeyPairs",
"ec2:RunInstances",
"ec2:DeleteKeyPair",
"ec2:DescribeVpcAttribute",
"ec2:CreateTags",
"ec2:DescribeSubnets",
"ec2:TerminateInstances",
"ec2:DescribeSecurityGroups",
"ec2:DescribeImages",
"ec2:DescribeVpcs",
"ec2:CreateVpc",
"ec2:CreateSubnet",
"ec2:DescribeAvailabilityZones",
"ec2:CreateRoute",
"ec2:CreateInternetGateway",
"ec2:AttachInternetGateway",
"ec2:ModifyVpcAttribute",
"ec2:CreateSecurityGroup",
"ec2:DeleteSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:DescribeRouteTables",
"ec2:DescribeInstanceTypes"
],
"Resource": "*"
}
]
}
```
# Copy data from an Amazon S3 bucket to another account and Region by using the AWS CLI
*Appasaheb Bagali and Purushotham G K, Amazon Web Services*
## Summary
This pattern describes how to migrate data from a source Amazon Simple Storage Service (Amazon S3) bucket in an AWS account to a destination Amazon S3 bucket in another AWS account, either in the same AWS Region or in a different Region.
The source Amazon S3 bucket allows AWS Identity and Access Management (IAM) access by using an attached resource policy. A user in the destination account has to assume a role that has `PutObject` and `GetObject` permissions for the source bucket. Finally, you run `copy` and `sync` commands to transfer data from the source Amazon S3 bucket to the destination Amazon S3 bucket.
Accounts own the objects that they upload to Amazon S3 buckets. If you copy objects across accounts and Regions, you grant the destination account ownership of the copied objects. You can change the ownership of an object by changing its [access control list (ACL)](https://docs.aws.amazon.com/AmazonS3/latest/dev/S3_ACLs_UsingACLs.html) to `bucket-owner-full-control`. However, we recommend that you grant programmatic cross-account permissions to the destination account because ACLs can be difficult to manage for multiple objects.
**Warning**
This scenario requires IAM users with programmatic access and long-term credentials, which present a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. Access keys can be updated if necessary. For more information, see [Updating access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) in the IAM documentation.
## Prerequisites and limitations
*Prerequisites*
+ Two active AWS accounts in the same or different AWS Regions.
+ An existing Amazon S3 bucket in the source account.
+ If your source or destination Amazon S3 bucket has [default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) enabled, you must modify the AWS Key Management Service (AWS KMS) key permissions. For more information, see the [AWS re:Post article](https://repost.aws/knowledge-center/s3-bucket-access-default-encryption) on this topic.
+ Familiarity with cross-account permissions.
*Limitations*
+ This pattern covers one-time migration. For scenarios that require continuous and automatic migration of new objects from a source bucket to a destination bucket, you can use [Amazon S3 Batch Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html).
+ This patterns uses session credentials (`AccessKeyId`, `SecretAccessKey`, and `SessionToken`) that are temporary and non-persistent. The expiration timestamp in the output indicates when these credentials expire. The role is configured with the maximum session duration. The copy job will be canceled if the session expires.
## Architecture
![\[Copying Amazon S3 data to another account or Region\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a574c26b-fdd9-4472-842b-b34c3eb2bfe9/images/5e4dec53-dfc8-478b-a7c4-503d63c8ac4e.png)
## Tools
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command line shell.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
## Best practices
+ [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) (IAM documentation)
+ [Applying least-privilege permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) (IAM documentation)
## Epics
### Create an IAM user and role in the destination AWS account
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM user and get the access key. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-data-from-an-s3-bucket-to-another-account-and-region-by-using-the-aws-cli.html) | AWS DevOps |
| Create an IAM identity-based policy. | Create an IAM identity-based policy named `S3MigrationPolicy` by using the following permissions. Modify the source and destination bucket names according to your use case. This identity-based policy allows the user who is assuming this role to access the source bucket and destination bucket. For detailed instructions, see [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html) in the IAM documentation. {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListObjectsV2",
"s3:GetObject",
"s3:GetObjectTagging",
"s3:GetObjectVersion",
"s3:GetObjectVersionTagging"
],
"Resource": [
"arn:aws:s3:::amazon-s3-demo-source-bucket",
"arn:aws:s3:::amazon-s3-demo-source-bucket/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging",
"s3:GetObjectTagging",
"s3:ListObjectsV2",
"s3:GetObjectVersion",
"s3:GetObjectVersionTagging"
],
"Resource": [
"arn:aws:s3:::amazon-s3-demo-destination-bucket",
"arn:aws:s3:::amazon-s3-demo-destination-bucket/*"
]
}
]
} | AWS DevOps |
| Create an IAM role. | Create an IAM role named `S3MigrationRole` by using the following trust policy. Modify the Amazon Resource Name (ARN) of the destination IAM role or user name in the trust policy according to your use case. This trust policy allows the newly created IAM user to assume `S3MigrationRole`. Attach the previously created `S3MigrationPolicy`. For detailed steps, see [Creating a role to delegate permissions to an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) in the IAM documentation.{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam:::user/"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
} | AWS DevOps |
### Create and attach the Amazon S3 bucket policy in the source account
| Task | Description | Skills required |
| --- | --- | --- |
| Create and attach an Amazon S3 bucket policy. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-data-from-an-s3-bucket-to-another-account-and-region-by-using-the-aws-cli.html) | Cloud administrator |
### Configure the destination Amazon S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Create a destination Amazon S3 bucket. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-data-from-an-s3-bucket-to-another-account-and-region-by-using-the-aws-cli.html) | Cloud administrator |
### Copy data to the destination Amazon S3 bucket
| Task | Description | Skills required |
| --- | --- | --- |
| Configure the AWS CLI with the newly created user credentials. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-data-from-an-s3-bucket-to-another-account-and-region-by-using-the-aws-cli.html) | AWS DevOps |
| Assume the Amazon S3 migration role. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-data-from-an-s3-bucket-to-another-account-and-region-by-using-the-aws-cli.html)For more information, see [How do I use the AWS CLI to assume an IAM role?](https://repost.aws/knowledge-center/iam-assume-role-cli) | AWS administrator |
| Copy and synchronize data from the source bucket to the destination bucket. | When you have assumed the role `S3MigrationRole` you can copy the data using the [copy](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) (`cp`) or [synchronize](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) (`sync`) command.Copy:aws s3 cp s3://amazon-s3-demo-source-bucket/ \
s3://amazon-s3-demo-destination-bucket/ \
--recursive --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME
Synchronize:aws s3 sync s3://amazon-s3-demo-source-bucket/ \
s3://amazon-s3-demo-destination-bucket/ \
--source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME
| Cloud administrator |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| An error occurred (`AccessDenied`) when calling the `ListObjects` operation | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-data-from-an-s3-bucket-to-another-account-and-region-by-using-the-aws-cli.html) |
## Related resources
+ [Creating an Amazon S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) (Amazon S3 documentation)
+ [Amazon S3 bucket policies and user policies](https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html) (Amazon S3 documentation)
+ [IAM identities (users, groups, and roles)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html?icmpid=docs_iam_console) (IAM documentation)
+ [cp command](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) (AWS CLI documentation)
+ [sync command](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) (AWS CLI documentation)
# Enable DB2 log archiving directly to Amazon S3 in an IBM Db2 database
*Ambarish Satarkar, Amazon Web Services*
## Summary
This pattern describes how to use Amazon Simple Storage Service (Amazon S3) as catalog storage for archive logs that are generated by IBM Db2, without using a staging area.
You can specify [DB2REMOTE](https://www.ibm.com/docs/en/db2/12.1.0?topic=storage-db2remote-identifiers) Amazon S3 storage for the [logarchmeth1](https://www.ibm.com/docs/en/db2/12.1.0?topic=parameters-logarchmeth1-primary-log-archive-method) and [logarchmeth2](https://www.ibm.com/docs/en/db2/12.1.0?topic=parameters-logarchmeth2-secondary-log-archive-method) log archive method configuration parameters. You can use the `logarchmeth1` parameter to specify the primary destination for logs that are archived from the current log path. With this capability, you can archive and retrieve transaction logs to and from Amazon S3 directly, without using a staging area.
[Amazon S3](https://aws.amazon.com/s3/) stores the data that’s uploaded to it across at least three devices in a single AWS Region. Millions of customers of all sizes and industries use Amazon S3 for storing enterprise backups given its high availability, flexible storage options, lifecycle policies, and security.
## Prerequisites and limitations
**Prerequisites **
+ An active AWS account.
+ IBM Db2 database running on an Amazon Elastic Compute Cloud (Amazon EC2) instance.
+ AWS Command Line Interface (AWS CLI) installed
+ [libcurl](https://curl.se/libcurl/) and [libxml2](https://gitlab.gnome.org/GNOME/libxml2/-/wikis/home) installed on the Db2 EC2 instance.
**Limitations **
+ Only [Db2 11.5.7](https://www.ibm.com/docs/en/db2/11.5.x?topic=new-1157) or later allows log archiving directly to Amazon S3 storage.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
+ In all configurations, the following limitations exist for Amazon S3:
+ AWS Key Management Service (AWS KMS) is not supported.
+ AWS role-based (AWS Identity and Access Management (IAM)) or token-based (AWS Security Token Service (AWS STS)) credentials are not supported.
**Product versions**
+ AWS CLI version 2 or later
+ IBM Db2 11.5.7 or later
+ Linux SUSE Linux Enterprise Server (SLES) 11 or later
+ Red Hat Enterprise Linux (RHEL) 6 or later
+ Windows Server 2008 R2, 2012 (R2), 2016, or 2019
## Architecture
The following diagram shows the components and workflow for this pattern.
![\[Workflow to use Amazon S3 for catalog storage for archive logs generated by Db2.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/7a10333e-07be-4144-9913-45c60a2f51ea/images/0437d348-1688-4c3e-9aa5-43535afe08c6.png)
The architecture on the AWS Cloud includes the following:
+ **Virtual private cloud (VPC)** – A logically isolated section of the AWS Cloud where you launch resources.
+ **Availability Zone** – Provides high availability by running the Db2 LUW (Linux, Unix, Windows) workload in an isolated data center within the AWS Region.
+ **Public subnet** – Provides RDP (Remote Desktop Protocol) access for administrators and internet connectivity through a NAT gateway.
+ **Private subnet** – Hosts the Db2 LUW database. The Db2 LUW instance is configured with the `LOGARCHMETH1` parameter. The parameter writes database log archive files directly to an Amazon S3 path through the gateway endpoint.
The following AWS services provide support:
+ **Amazon S3** – Serves as the durable, scalable storage location for Db2 log archive files.
+ **Amazon Elastic File System (Amazon EFS)** – Provides a shared, fully managed file system that Db2 can use for database backups and staging. Db2 can also use Amazon EFS as a mount point for log files before they are archived to Amazon S3.
+ **Amazon CloudWatch** – Collects and monitors metrics, logs, and events from Db2 and the underlying EC2 instances. You can use CloudWatch to create alarms, dashboards, and automated responses to performance or availability issues.
**Automation and scale**
+ This pattern provides a fully automated solution to store Db2 log archive backup.
+ You can use the same Amazon S3 bucket to enable log archive of multiple Db2 databases.
## Tools
**AWS services**
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) helps you monitor the metrics of your AWS resources and the applications you run on AWS in real time.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open source tool that helps you interact with AWS services through commands in your command-line shell.
+ [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud.
+ [AWS IAM Identity Center](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) helps you centrally manage single sign-on (SSO) access to all of your AWS accounts and cloud applications.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
**Other tools**
+ [libcurl](https://curl.se/libcurl/) is a free client-side URL transfer library.
+ [libxml2](https://gitlab.gnome.org/GNOME/libxml2/-/wikis/home) is a free XML C parser and toolkit.
## Best practices
+ Follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation.
## Epics
### Configure AWS services
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the AWS CLI. | To [download and install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.htmlhttps://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), use the following commands:i) curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
ii) unzip awscliv2.zip
iii) sudo ./aws/install
| AWS systems administrator, AWS administrator |
| Configure the AWS CLI. | To [configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html), use the following commands:$ aws configure
AWS Access Key ID [None]:*******************************
AWS Secret Access Key [None]: ***************************
Default region name [None]: < aws region >
Default output format [None]: text
| AWS systems administrator, AWS administrator |
| Create IAM user. | To create an IAM user to use later for the Db2 database connection with Amazon S3, use the following command:`aws iam create-user --user-name `Following is an example of the command:`aws iam create-user --user-name db_backup_user`This scenario requires IAM users with programmatic access and long-term credentials, which presents a security risk. To mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. Access keys can be updated if necessary. For more information, see [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys) and [Manage access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) in the IAM documentation. | AWS systems administrator |
| Create Amazon S3 bucket. | To create an Amazon S3 bucket for storing the database backup, use the following command:`aws s3api create-bucket --bucket --region `Following is an example command:`aws s3api create-bucket --bucket myfirstbucket --region af-south-1` | AWS systems administrator |
| Authorize the IAM user. | To authorize the newly created IAM user to have Amazon S3 permissions, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/enable-db2-logarchive-directly-to-amazon-s3-in-ibm-db2-database.html) | AWS systems administrator, AWS administrator |
| Create access key. | To generate an access key to programmatically access Amazon S3 from the DB2 instance, use the following command:`aws iam create-access-key --user-name `Following is an example of the command:`aws iam create-access-key --user-name db_backup_user`This scenario requires IAM users with programmatic access and long-term credentials, which presents a security risk. To mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed. Access keys can be updated if necessary. For more information, see [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys) and [Manage access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) in the IAM documentation. | AWS systems administrator |
| Create a PKCS keystore. | To create a PKCS keystore to store the key and create a secret access key to transfer the data to Amazon S3, use the following command: gsk8capicmd_64 -keydb -create -db "/db2/db2/.keystore/db6-s3.p12" -pw "" -type pkcs12 -stash
| AWS systems administrator |
| Configure DB2 to use the keystore. | To configure DB2 to use the keystore with the `keystore_location` and `keystore_type` parameters, use the following commands:db2 "update dbm cfg using keystore_location /db2/db2/.keystore/db6-s3.p12 keystore_type pkcs12"
| AWS systems administrator |
| Create a DB2 storage access alias. | A storage access alias specifies the Amazon S3 bucket to use. It also provides the connection details such as the username and password that are stored in the local keystore in an encrypted format. For more information, see [CATALOG STORAGE ACCESS command](https://www.ibm.com/docs/en/db2/12.1.0?topic=commands-catalog-storage-access) in the IBM Db2 documentation.To create a storage access alias, use the following syntax:db2 "catalog storage access alias vendor S3 server user '' password '' container ''"
Following is an example:db2 "catalog storage access alias DB2BKPS3 vendor S3 server s3.us-west-2.amazonaws.com user '*******************' password '*********************' container 'myfirstbucket'"
| AWS systems administrator |
### Update logarchmeth1 location in DB2 and restart DB2
| Task | Description | Skills required |
| --- | --- | --- |
| Update the `LOGARCHMETH1` location. | To use the storage access alias that you defined earlier, update the `LOGARCHMETH1` database parameters, use the following command:db2 update db cfg for using LOGARCHMETH1 'DB2REMOTE:////'
To separate the logs from other files, specify a subdirectory (that is, the Amazon S3 bucket prefix) `TESTDB_LOGS` in which to save the logs within the S3 bucket.Following is an example:db2 update db cfg for ABC using LOGARCHMETH1 'DB2REMOTE://DB2BKPS3//TESTDB_LOGS/'
You should see the following message: `DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.` | AWS systems administrator |
| Restart DB2. | Restart the DB2 instance after reconfiguring it for log archiving.However, if `LOGARCHMETH1 `was previously set to any file system location, then restart is not required. | AWS administrator, AWS systems administrator |
### Check the archive log path in Amazon S3 and db2diag.log
| Task | Description | Skills required |
| --- | --- | --- |
| Check the archive log in Amazon S3. | At this point, your database is completely configured to archive the transaction logs directly to the Amazon S3 storage. To confirm the configuration, start executing transactional activities on the database to start consuming (and archiving) the log space. Then, check the archive logs in Amazon S3. | AWS administrator, AWS systems administrator |
| Check archive log configuration in `db2diag.log`. | After you check the archive log in Amazon S3, look for the following message in the DB2 diagnostic log `db2diag.log`:`MESSAGE : ADM1846I Completed archive for log file "S0000079.LOG" to Completed archive for log file S0000080.LOG to DB2REMOTE:////log1/db2//NODE0000/LOGSTREAM0000/C0000001/ from /db2//log_dir/NODE0000/LOGSTREAM0000/. MESSAGE : ADM1846I Completed archive for log file "S0000080.LOG" to Completed archive for log file S0000081.LOG to DB2REMOTE:// //log1/db2//NODE0000/LOGSTREAM0000/C0000001/ from /db2//log_dir/NODE0000/LOGSTREAM0000/. `This message confirms that the closed DB2 transaction log files are being archived to the (remote) Amazon S3 storage. | AWS systems administrator |
## Related resources
**AWS service documentation**
+ [AWS security credentials ](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys)(IAM documentation)
+ [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) (IAM documentation)
+ [Manage access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) (IAM documentation)
+ [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) (IAM documentation)
**IBM resources**
+ [IBM Db2 Database](https://www.ibm.com/products/db2-database)
+ [logarchmeth1 - Primary log archive method configuration parameter](https://www.ibm.com/docs/en/db2/12.1.0?topic=parameters-logarchmeth1-primary-log-archive-method)
+ [logarchmeth2 - Secondary log archive method configuration parameter](https://www.ibm.com/docs/en/db2/12.1.0?topic=parameters-logarchmeth2-secondary-log-archive-method)
+ [Remote storage](https://www.ibm.com/docs/en/db2/12.1.0?topic=databases-remote-storage)
# Migrate data from an on-premises Hadoop environment to Amazon S3 using DistCp with AWS PrivateLink for Amazon S3
*Jason Owens, Andres Cantor, Jeff Klopfenstein, Bruno Rocha Oliveira, and Samuel Schmidt, Amazon Web Services*
## Summary
This pattern demonstrates how to migrate nearly any amount of data from an on-premises Apache Hadoop environment to the Amazon Web Services (AWS) Cloud by using the Apache open-source tool [DistCp](https://hadoop.apache.org/docs/r1.2.1/distcp.html) with AWS PrivateLink for Amazon Simple Storage Service (Amazon S3). Instead of using the public internet or a proxy solution to migrate data, you can use [AWS PrivateLink for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) to migrate data to Amazon S3 over a private network connection between your on-premises data center and an Amazon Virtual Private Cloud (Amazon VPC). If you use DNS entries in Amazon Route 53 or add entries in the **/etc/hosts** file in all nodes of your on-premises Hadoop cluster, then you are automatically directed to the correct interface endpoint.
This guide provides instructions for using DistCp for migrating data to the AWS Cloud. DistCp is the most commonly used tool, but other migration tools are available. For example, you can use offline AWS tools like [AWS Snowball](https://docs.aws.amazon.com/whitepapers/latest/how-aws-pricing-works/aws-snow-family.html#aws-snowball) or [AWS Snowmobile](https://docs.aws.amazon.com/whitepapers/latest/how-aws-pricing-works/aws-snow-family.html#aws-snowmobile), or online AWS tools like [AWS Storage Gateway](https://docs.aws.amazon.com/storagegateway/latest/userguide/migrate-data.html) or [AWS DataSync](https://aws.amazon.com/about-aws/whats-new/2021/11/aws-datasync-hadoop-aws-storage-services/). Additionally, you can use other open-source tools like [Apache NiFi](https://nifi.apache.org/).
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account with a private network connection between your on-premises data center and the AWS Cloud
+ [Hadoop](https://hadoop.apache.org/releases.html), installed on premises with [DistCp](https://hadoop.apache.org/docs/r1.2.1/distcp.html)
+ A Hadoop user with access to the migration data in the Hadoop Distributed File System (HDFS)
+ AWS Command Line Interface (AWS CLI), [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
+ [Permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket-console.html) to put objects into an S3 bucket
**Limitations**
Virtual private cloud (VPC) limitations apply to AWS PrivateLink for Amazon S3. For more information, see [Interface endpoint properties and limitations](https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-interface.html#vpce-interface-limitations) and [AWS PrivateLink quotas](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-limits-endpoints.html) (AWS PrivateLink documentation).
AWS PrivateLink for Amazon S3 doesn't support the following:
+ [Federal Information Processing Standard (FIPS) endpoints](https://aws.amazon.com/compliance/fips/)
+ [Website endpoints](https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteEndpoints.html)
+ [Legacy global endpoints](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#deprecated-global-endpoint)
## Architecture
**Source technology stack**
+ Hadoop cluster with DistCp installed
**Target technology stack**
+ Amazon S3
+ Amazon VPC
**Target architecture**
![\[Hadoop cluster with DistCp copies data from on-premises environment through Direct Connect to S3.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/8d2b47ae-e854-4e5d-8f19-b9c2606f2c59/images/b8a249bd-307b-41ec-b939-5039d0ae7123.png)
The diagram shows how the Hadoop administrator uses DistCp to copy data from an on-premises environment through a private network connection, such as AWS Direct Connect, to Amazon S3 through an Amazon S3 interface endpoint.
## Tools
**AWS services**
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
**Other tools**
+ [Apache Hadoop DistCp](https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html) (distributed copy) is a tool used for copying large inter-clusters and intra-clusters. DistCp uses Apache MapReduce for distribution, error handling and recovery, and reporting.
## Epics
### Migrate data to the AWS Cloud
| Task | Description | Skills required |
| --- | --- | --- |
| Create an endpoint for AWS PrivateLink for Amazon S3. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-data-from-an-on-premises-hadoop-environment-to-amazon-s3-using-distcp-with-aws-privatelink-for-amazon-s3.html) | AWS administrator |
| Verify the endpoints and find the DNS entries. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-data-from-an-on-premises-hadoop-environment-to-amazon-s3-using-distcp-with-aws-privatelink-for-amazon-s3.html) | AWS administrator |
| Check the firewall rules and routing configurations. | To confirm that your firewall rules are open and that your networking configuration is correctly set up, use Telnet to test the endpoint on port 443. For example:$ telnet vpce-.s3.us-east-2.vpce.amazonaws.com 443
Trying 10.104.88.6...
Connected to vpce-.s3.us-east-2.vpce.amazonaws.com.
...
$ telnet vpce-.s3.us-east-2.vpce.amazonaws.com 443
Trying 10.104.71.141...
Connected to vpce-.s3.us-east-2.vpce.amazonaws.com.
If you use the Regional entry, a successful test shows that the DNS is alternating between the two IP addresses that you can see on the **Subnets** tab for your selected endpoint in the Amazon VPC console. | Network administrator, AWS administrator |
| Configure the name resolution. | You must configure the name resolution to allow Hadoop to access the Amazon S3 interface endpoint. You can’t use the endpoint name itself. Instead, you must resolve `.s3..amazonaws.com` or `*.s3..amazonaws.com`. For more information on this naming limitation, see [Introducing the Hadoop S3A client](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Introducing_the_Hadoop_S3A_client.) (Hadoop website).Choose one of the following configuration options:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-data-from-an-on-premises-hadoop-environment-to-amazon-s3-using-distcp-with-aws-privatelink-for-amazon-s3.html) | AWS administrator |
| Configure authentication for Amazon S3. | To authenticate to Amazon S3 through Hadoop, we recommend that you export temporary role credentials to the Hadoop environment. For more information, see [Authenticating with S3](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authenticating_with_S3) (Hadoop website). For long-running jobs, you can create a user and assign a policy that has permissions to put data into an S3 bucket only. The access key and secret key can be stored on Hadoop, accessible only to the DistCp job itself and to the Hadoop administrator. For more information on storing secrets, see [Storing secrets with Hadoop Credential Providers](https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/index.html#hadoop_credential_providers) (Hadoop website). For more information on other authentication methods, see [How to get credentials of an IAM role for use with CLI access to an AWS account](https://docs.aws.amazon.com/singlesignon/latest/userguide/howtogetcredentials.html) in the documentation for AWS IAM Identity Center (successor to AWS Single Sign-On).To use temporary credentials, add the temporary credentials to your credentials file, or run the following commands to export the credentials to your environment:export AWS_SESSION_TOKEN=SECRET-SESSION-TOKEN
export AWS_ACCESS_KEY_ID=SESSION-ACCESS-KEY
export AWS_SECRET_ACCESS_KEY=SESSION-SECRET-KEY
If you have a traditional access key and secret key combination, run the following commands:export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key
If you use an access key and secret key combination, then change the credentials provider in the DistCp commands from `"org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider"` to `"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"`. | AWS administrator |
| Transfer data by using DistCp. | To use DistCp to transfer data, run the following commands:hadoop distcp -Dfs.s3a.aws.credentials.provider=\
"org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" \
-Dfs.s3a.access.key="${AWS_ACCESS_KEY_ID}" \
-Dfs.s3a.secret.key="${AWS_SECRET_ACCESS_KEY}" \
-Dfs.s3a.session.token="${AWS_SESSION_TOKEN}" \
-Dfs.s3a.path.style.access=true \
-Dfs.s3a.connection.ssl.enabled=true \
-Dfs.s3a.endpoint=s3..amazonaws.com \
hdfs:///user/root/ s3a://
The AWS Region of the endpoint isn’t automatically discovered when you use the DistCp command with AWS PrivateLink for Amazon S3. Hadoop 3.3.2 and later versions resolve this issue by enabling the option to explicitly set the AWS Region of the S3 bucket. For more information, see [S3A to add option fs.s3a.endpoint.region to set AWS region](https://issues.apache.org/jira/browse/HADOOP-17705) (Hadoop website).For more information on additional S3A providers, see [General S3A Client configuration](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration) (Hadoop website). For example, if you use encryption, you can add the following option to the series of commands above depending on your type of encryption:-Dfs.s3a.server-side-encryption-algorithm=AES-256 [or SSE-C or SSE-KMS]
To use the interface endpoint with S3A, you must create a DNS alias entry for the S3 Regional name (for example, `s3..amazonaws.com`) to the interface endpoint. See the *Configure authentication for Amazon S3* section for instructions. This workaround is required for Hadoop 3.3.2 and earlier versions. Future versions of S3A won’t require this workaround.If you have signature issues with Amazon S3, add an option to use Signature Version 4 (SigV4) signing:-Dmapreduce.map.java.opts="-Dcom.amazonaws.services.s3.enableV4=true"
| Migration engineer, AWS administrator |
# More patterns
**Topics**
+ [Access AWS services from IBM z/OS by installing the AWS CLI](access-aws-services-from-ibm-z-os-by-installing-aws-cli.md)
+ [Automatically archive items to Amazon S3 using DynamoDB TTL](automatically-archive-items-to-amazon-s3-using-dynamodb-ttl.md)
+ [Automatically back up SAP HANA databases using Systems Manager and EventBridge](automatically-back-up-sap-hana-databases-using-systems-manager-and-eventbridge.md)
+ [Back up and archive mainframe data to Amazon S3 using BMC AMI Cloud Data](back-up-and-archive-mainframe-data-to-amazon-s3-using-bmc-ami-cloud-data.md)
+ [Build an ETL service pipeline to load data incrementally from Amazon S3 to Amazon Redshift using AWS Glue](build-an-etl-service-pipeline-to-load-data-incrementally-from-amazon-s3-to-amazon-redshift-using-aws-glue.md)
+ [Configure model invocation logging in Amazon Bedrock by using AWS CloudFormation](configure-bedrock-invocation-logging-cloudformation.md)
+ [Convert and unpack EBCDIC data to ASCII on AWS by using Python](convert-and-unpack-ebcdic-data-to-ascii-on-aws-by-using-python.md)
+ [Convert VARCHAR2(1) data type for Oracle to Boolean data type for Amazon Aurora PostgreSQL](convert-varchar2-1-data-type-for-oracle-to-boolean-data-type-for-amazon-aurora-postgresql.md)
+ [Create an Amazon ECS task definition and mount a file system on EC2 instances using Amazon EFS](create-an-amazon-ecs-task-definition-and-mount-a-file-system-on-ec2-instances-using-amazon-efs.md)
+ [Deliver DynamoDB records to Amazon S3 using Kinesis Data Streams and Firehose with AWS CDK](deliver-dynamodb-records-to-amazon-s3-using-kinesis-data-streams-and-amazon-data-firehose-with-aws-cdk.md)
+ [Deploy a Lustre file system for high-performance data processing by using Terraform and DRA](deploy-lustre-file-system-for-high-performance-data-processing-with-terraform-dra.md)
+ [Estimate storage costs for an Amazon DynamoDB table](estimate-storage-costs-for-an-amazon-dynamodb-table.md)
+ [Identify public Amazon S3 buckets in AWS Organizations by using Security Hub CSPM](identify-public-s3-buckets-in-aws-organizations-using-security-hub.md)
+ [Migrate an on-premises SFTP server to AWS using AWS Transfer for SFTP](migrate-an-on-premises-sftp-server-to-aws-using-aws-transfer-for-sftp.md)
+ [Migrate an Oracle partitioned table to PostgreSQL by using AWS DMS](migrate-an-oracle-partitioned-table-to-postgresql-by-using-aws-dms.md)
+ [Migrate data from Microsoft Azure Blob to Amazon S3 by using Rclone](migrate-data-from-microsoft-azure-blob-to-amazon-s3-by-using-rclone.md)
+ [Migrate Oracle CLOB values to individual rows in PostgreSQL on AWS](migrate-oracle-clob-values-to-individual-rows-in-postgresql-on-aws.md)
+ [Migrate shared file systems in an AWS large migration](migrate-shared-file-systems-in-an-aws-large-migration.md)
+ [Migrate small sets of data from on premises to Amazon S3 using AWS SFTP](migrate-small-sets-of-data-from-on-premises-to-amazon-s3-using-aws-sftp.md)
+ [Monitor Amazon Aurora for instances without encryption](monitor-amazon-aurora-for-instances-without-encryption.md)
+ [Move mainframe files directly to Amazon S3 using Transfer Family](move-mainframe-files-directly-to-amazon-s3-using-transfer-family.md)
+ [Remove Amazon EC2 entries in the same AWS account from AWS Managed Microsoft AD by using AWS Lambda automation](remove-amazon-ec2-entries-in-the-same-aws-account-from-aws-managed-microsoft-ad.md)
+ [Run stateful workloads with persistent data storage by using Amazon EFS on Amazon EKS with AWS Fargate](run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate.md)
+ [Successfully import an S3 bucket as an AWS CloudFormation stack](successfully-import-an-s3-bucket-as-an-aws-cloudformation-stack.md)
+ [Synchronize data between Amazon EFS file systems in different AWS Regions by using AWS DataSync](synchronize-data-between-amazon-efs-file-systems-in-different-aws-regions-by-using-aws-datasync.md)
+ [View EBS snapshot details for your AWS account or organization](view-ebs-snapshot-details-for-your-aws-account-or-organization.md)