Creating an object storage location for AWS DataSync - AWS DataSync

Creating an object storage location for AWS DataSync

A location is an endpoint for an object storage system, which can be hosted on-premises or by another cloud provider (for example, a Google Cloud Storage bucket). AWS DataSync can use the location as a source or destination for copying data.

Prerequisites

Your object storage system must be compatible with the following Amazon S3 API operations for AWS DataSync to connect to it:

  • AbortMultipartUpload

  • CompleteMultipartUpload

  • CopyObject

  • CreateMultipartUpload

  • DeleteObject

  • DeleteObjects

  • DeleteObjectTagging

  • GetBucketLocation

  • GetObject

  • GetObjectTagging

  • HeadBucket

  • HeadObject

  • ListObjectsV2

  • PutObject

  • PutObjectTagging

  • UploadPart

Your object storage system must also support AWS Signature Version 4 for authenticating requests. (AWS Signature Version 2 is deprecated.)

Considerations when migrating to or from a Google Cloud Storage bucket

Because DataSync communicates with Google Cloud Storage by using the Amazon S3 API, there's a limitation that might cause your DataSync task to fail if you try to copy object tags. To prevent this, clear the Copy object tags check box when configuring your task settings. For more information, see Managing how AWS DataSync transfers files, objects, and metadata.

For detailed instructions on migrating from Google Cloud Storage, see Tutorial: Transferring data from Google Cloud Storage to Amazon S3.

Creating the location

Your object storage system can be a source or destination location for DataSync.

To create an object storage location by using the console
  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, choose Locations.

  3. On the Locations page, choose Create location.

  4. For Location type, choose Object storage.

    You configure this location as a source or destination later.

  5. For Agents, choose one or more DataSync agents.

    During the transfer, the agents securely connect to your object storage server.

  6. For Server, provide the domain name or IP address of the object storage server.

  7. For Bucket name, enter the name of the object storage bucket involved in the transfer.

  8. For Folder, enter an object prefix.

    DataSync only copies objects with this prefix.

  9. To configure the connection to the object storage server, expand Additional settings and do the following:

    1. For Server protocol, choose HTTP or HTTPS.

    2. For Server port, use a default port (80 for HTTP or 443 for HTTPS) or specify a custom port if needed.

    3. For Certificate, select Choose file to specify a private or self-signed certificate that the DataSync agent will trust when connecting to your object storage server.

      The custom certificate file must have a .pem extension.

  10. If credentials are required to access the object storage server, select Requires credentials and enter the Access key and Secret key for accessing the bucket.

    You can use the access key and secret key settings to provide a user name and password, respectively.

  11. Select Add tag to tag your location.

    Tags help you manage, filter, and search for your AWS resources. We recommend creating at least a name tag for your location.

  12. Choose Create location.