AWS DataSync
User Guide

Creating a Location for Amazon S3

A location for Amazon S3 is an endpoint for the Amazon S3 bucket that DataSync uses as a source or destination.

To create an S3 location

  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the navigation pane, choose Locations. The locations that you previously created appear in the list of locations.

  3. On the Create location page, choose Create location.

  4. For Location type, choose Amazon S3 bucket.

  5. For S3 Bucket, choose the S3 bucket that you want to use as an endpoint. You configure this location as a source or destination later.

  6. For S3 storage class, choose a storage class you want to transfer objects directly into. For information about storage classes, see Amazon S3 Storage Classes in the Amazon Simple Storage Service Developer Guide. Some storage classes have behaviors that can affect your S3 cost. For detailed information, see Considerations When Working with S3 Storage Classes in DataSync.

  7. For Folder, provide the name of a folder in S3. This is the S3 folder that DataSync uses, either to read data from for an S3 source location or write data to for an S3 destination.

  8. For IAM role, choose Autogenerate for DataSync to automatically create a role with the required permissions.

    If DataSync has previously created such a role for this S3 bucket, that role is chosen as the default in the list. You can also create your own role and choose it from the list. For instructions on how to create an IAM role manually, see Manually Configuring an IAM Role to Access Your S3 Bucket .

  9. (Optional) For Key and Value, provide values to tag your S3 location. A tag is a key-value pair that helps you manage, filter, and search for your locations.

  10. When you are done, choose Create location. The location that you just created appears in the list of locations.

Note

In addition to the IAM policies that grant DataSync permissions, we recommend creating a multipart upload bucket policy for your S3 buckets to help you control your storage cost. For more information, see the blog post S3 Lifecycle Management Update – Support for Multipart Uploads and Delete Markers.

Amazon S3 Location Settings

If the location you want to use as a source or a destination is an Amazon S3 bucket, you configure the following settings.

S3 bucket

The Amazon S3 bucket that you want to use as a source or destination location.

Storage class

This enables you to transfer objects directly into the Amazon S3 storage class that you choose. For information about storage classes, see Amazon S3 Storage Classes in the Amazon Simple Storage Service Developer Guide. Some storage classes have specific behaviors that can affect your S3 storage cost. For detailed information, see Considerations When Working with S3 Storage Classes in DataSync.

Folder

This is the S3 folder that DataSync uses, either to read data from for an S3 source location or write data to for an S3 destination.

IAM role

The AWS Identity and Access Management (IAM) role that has permissions to access the S3 bucket.

For DataSync to access a destination S3 bucket, it requires access to your Amazon S3 bucket. To obtain this access, DataSync assumes the IAM role that you provide. The role requires an IAM policy and a security token service trust (STS) relationship. The policy determines which actions the role can perform. DataSync can create the role on your behalf. For instructions, see Creating a Location for Amazon S3. You can also create the role manually and choose it from the list in the console. For instructions, see Manually Configuring an IAM Role to Access Your S3 Bucket .

Tag

A key-value pair that identifies the S3 location. By default, the DataSync console prepopulates a name value with the task or location name.

Considerations When Working with S3 Storage Classes in DataSync

DataSync can transfer objects directly into the S3 storage class that you choose. For more information about S3 storage classes, see Amazon S3 Storage Classes. Some storage classes have behaviors that can affect your S3 storage cost. For more information, see Amazon S3 Pricing.

Following, you can find some considerations for how each S3 storage class works with DataSync.

S3 Storage Class Considerations
Standard Choose Standard to store your frequently accessed files redundantly in multiple Availability Zones that are geographically separated. This is the default if you don't specify a storage class.
Intelligent-Tiering

Choose Intelligent-Tiering to optimize storage costs by automatically moving data to the most cost-effective storage access tier.

Objects stored in the Intelligent-Tiering storage class can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them.

Objects less than 128 KB are not eligible for autotiering in the Intelligent-Tiering storage class. These objects are stored in Standard.

Standard-IA

Choose Standard-IA to store your infrequently accessed files redundantly in multiple Availability Zones that are geographically separated.

Objects stored in the Standard-IA storage class can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them.

Objects less than 128 KB are smaller than the minimum capacity charge per object in the Standard-IA storage class. These objects are stored in Standard.

One Zone-IA

Choose Standard-IA to store your infrequently accessed files in a single Availability Zone.

Objects stored in the One Zone-IA storage class can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them.

Objects less than 128 KB are smaller than the minimum capacity charge per object in the One Zone-IA storage class. These objects are stored in Standard.

Glacier

Choose Glacier to archive the files for more active archives.

Objects stored in Glacier can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them.

Objects less than 40 KB are smaller than the minimum capacity charge per object in the Glacier storage class. These objects are stored in Standard.

Glacier Deep Archive

Choose Glacier Deep Archive to archive your files for long-term data retention and digital preservation where data is accessed once or twice a year.

Objects stored in Glacier Deep Archive can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them.

Objects less than 40 KB are smaller than the minimum capacity charge per object in the Glacier Deep Archive storage class. These objects are stored in Standard.

Manually Configuring an IAM Role to Access Your S3 Bucket

When you use the DataSync Management Console to create an Amazon S3 location, DataSync automatically creates an IAM role that has the required permissions for you. If you want to create the IAM role manually, use the following procedure.

To manually configure an IAM role to access your S3 bucket

  1. Open the IAM Management Console.

  2. From the navigation pane, choose Roles, and then choose Create role to open the Create role page.

  3. In the Select type of trusted entity section, make sure that AWS service is selected.

  4. Under Choose the service that will use this role, choose DataSync, or manually configure it as shown in the following example.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
  5. Choose Next: Permissions.

  6. Choose AmazonS3FullAccess. You can also manually configure a more restricted policy. For an example of such a policy, see Amazon S3 Location Settings.

  7. (Optional) Choose Next: Tags to create tags for the role.

  8. Choose Next: Review, choose the role name, and then choose Create role.

  9. Open the DataSync Management Console.

  10. Choose the refresh button on the right side of the IAM role list, and then choose the role that you just created.