Creating an object storage location - AWS DataSync

Creating an object storage location

A location is an endpoint for an object storage system, which can be hosted on-premises or by another cloud provider (for example, a Google Cloud Storage bucket).

Prerequisites

Your object storage system must be compatible with the following Amazon S3 API operations for AWS DataSync to connect to it:

  • AbortMultipartUpload

  • CompleteMultipartUpload

  • CopyObject

  • CreateMultipartUpload

  • DeleteObject

  • DeleteObjects

  • DeleteObjectTagging

  • GetBucketLocation

  • GetObject

  • GetObjectTagging

  • HeadBucket

  • HeadObject

  • ListObjectsV2

  • PutObject

  • PutObjectTagging

  • UploadPart

Your object storage system must also support AWS Signature Version 4 for authenticating requests. (AWS Signature Version 2 is deprecated.)

Considerations when migrating to or from a Google Cloud Storage bucket

Because DataSync communicates with Google Cloud Storage by using the Amazon S3 API, there's a limitation that might cause your DataSync task to fail if you try to copy object tags. To prevent this, clear the Copy object tags check box when configuring your task settings. For more information, see File metadata and management options.

For detailed instructions on migrating from Google Cloud Storage, see Tutorial: Transferring data from Google Cloud Storage to Amazon S3.

Creating the location

Your object storage system can be a source or destination location for DataSync.

To create an object storage location

  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, choose Locations.

  3. On the Locations page, choose Create location.

  4. For Location type, choose Object storage.

    You configure this location as a source or destination later.

  5. For Agents, choose one or more agents that you want to use.

    During the transfer, the agent securely connects to your object storage server.

  6. For Server, provide the domain name or IP address of the object storage server.

  7. For Bucket name, enter the name of the object storage bucket involved in the transfer.

  8. For Folder, enter an object prefix.

    If this is a source location, DataSync only copies objects with this prefix. If this is a destination location, DataSync writes all objects with this prefix.

  9. To select the object storage server protocol and server port, choose Additional settings and select either HTTP or HTTPS.

    By default, the Server port is set to 80 for HTTP and 443 for HTTPS. You can specify a custom port if needed.

  10. (Optional) If credentials are required to access the object storage location, select Requires credentials and enter the Access key and Secret key for accessing the bucket.

    You can also use the access key and secret key settings to provide a user name and password, respectively.

  11. (Optional) Tags are key-value pairs that help you manage, filter, and search for your location.

    We recommend using tags for naming your locations.

  12. Choose Create location.