View a markdown version of this page

Configuring AWS DataSync transfers with Google Cloud Storage - AWS DataSync

Configuring AWS DataSync transfers with Google Cloud Storage

With AWS DataSync, you can transfer data between Google Cloud Storage and the following AWS storage services:

  • Amazon S3

  • Amazon EFS

  • Amazon FSx for Windows File Server

  • Amazon FSx for Lustre

  • Amazon FSx for OpenZFS

  • Amazon FSx for NetApp ONTAP

To begin the transfer setup, create a location for your Google Cloud Storage. This location can serve as either your transfer source or destination. Most transfers require a DataSync agent. Transfers between Google Cloud Storage and Amazon S3 using Enhanced mode do not require an agent. Use the agent that corresponds to your task mode.

Note

For private cloud connectivity between Google Cloud Storage and AWS, you need a DataSync agent.

Overview

DataSync uses the Google Cloud Storage XML API for data transfers. This API provides an Amazon S3 compatible interface for reading and writing data with Google Cloud Storage buckets.

When you use an agent for transfers, you can deploy it in Google Cloud Storage or your Amazon VPC.

When you use Basic mode for transfers, you need to authenticate to Google Cloud Storage using long-lived access and secret keys.

When you use Enhanced mode for transfer, DataSync has two options for authenticating to Google Cloud Storage: authentication with long-lived access key and secret key, or authentication with short-lived Google access token. The first option is discussed in detail below. The second option is discussed further in Federated identity (OIDC).

Agent in Google Cloud
  1. You deploy a DataSync agent in your Google Cloud environment.

  2. The agent reads your Google Cloud Storage bucket by using a Hash-based Message Authentication Code (HMAC) key.

  3. The objects from your Google Cloud Storage bucket transfer securely through TLS 1.3 into the AWS Cloud by using a public endpoint.

  4. The DataSync service writes the data to your S3 bucket.

The following diagram illustrates the transfer.

An example DataSync transfer shows how object data transfers from a Google Cloud Storage bucket to an S3 bucket. First, the DataSync agent is deployed in your Google Cloud environment. Then, the DataSync agent reads the Google Cloud Storage bucket. The data moves securely through a public endpoint into AWS, where DataSync writes the objects to an S3 bucket in the same AWS Region where you're using DataSync.
Agent in your VPC
  1. You deploy a DataSync agent in a virtual private cloud (VPC) in your AWS environment.

  2. The agent reads your Google Cloud Storage bucket by using a Hash-based Message Authentication Code (HMAC) key.

  3. The objects from your Google Cloud Storage bucket transfer securely through TLS 1.3 into the AWS Cloud by using a private VPC endpoint.

  4. The DataSync service writes the data to your S3 bucket.

The following diagram illustrates the transfer.

An example DataSync transfer shows how object data transfers from a Google Cloud Storage bucket to an S3 bucket. First, the DataSync agent is deployed in a VPC in AWS. Then, the DataSync agent reads the Google Cloud Storage bucket. The data moves securely through a VPC endpoint into AWS, where DataSync writes the objects to an S3 bucket in the same AWS Region as the VPC.

Costs

The fees associated with this migration might include:

Prerequisites

Before you begin, do the following if you haven’t already:

Creating an HMAC key for your Google Cloud Storage bucket

DataSync uses an HMAC key that's associated with your Google service account to authenticate with and read the bucket that you’re transferring data from. (For detailed instructions on how to create HMAC keys, see the Google Cloud Storage documentation.)

To create an HMAC key
  1. Create an HMAC key for your Google service account.

  2. Make sure that your Google service account has at least Storage Object Viewer permissions.

  3. Save your HMAC key's access ID and secret in a secure location.

    You'll need these items later to configure your DataSync source location.

Federated identity (OIDC)

Some third-party cloud storage object storage systems support use of temporary credentials to access the storage, and they also integrate with AWS Security Token Service (STS) to allow AWS services to obtain cloud storage access tokens using AWS credentials. This type of integration is often referred to as identity federation across cloud providers. DataSync supports identity federation with Google Cloud Storage.

Note

Federated identity access to Google Cloud Storage is supported only for Enhanced mode tasks. You can't create a Basic mode task that uses a location configured with federated identity.

For a location, provide either federated identity parameters (FederatedIdentity) or an access key and secret key (AccessKey and SecretKey), but not both.

Using temporary short-lived credentials is often a preferred alternative to using long-lived credentials. While long-lived credentials are usually private (and must be stored securely to assure privacy), short-lived temporary credentials can be obtained at runtime without having to leverage any private information. Cloud services can obtain and refresh the temporary credentials (access tokens) at runtime to assure uninterrupted access to cloud storage.

DataSync can access Google Cloud Storage (GCS) locations without having to store long-lived credentials. Instead, DataSync can obtain a short-lived signed AWS STS JWT (web token) and can leverage Google Workload Identity Federation to exchange the web token for an OpenId Connect (OIDC) access token via Google STS. The token exchange flow is as follows:

  • DataSync assumes your IAM role

  • DataSync calls AWS STS GetWebIdentityToken with assumed role's credentials to get a signed JWT

  • DataSync exchanges the JWT via Google STS for GCP access token

  • DataSync uses GCP access token to access GCS location

To configure GCS object storage location with federated identity parameters, follow these steps.

  1. Create AWS IAM role for DataSync with the following trust policy and permissions:

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "aws:SourceAccount": "CUSTOMER_AWS_ACCOUNT_ID" }, "ArnLike": { "aws:SourceArn": "arn:aws:datasync:REGION:CUSTOMER_AWS_ACCOUNT_ID:*" } } } ] }

    and

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:GetWebIdentityToken", "Resource": "*" } ] }
  2. Create a GCP Workload Identity Pool (or use an existing one):

    • Go to GCP IAM and Admin → Workload Identity Federation

    • Click Create Pool

    • Enter name and description, e.g. Name: datasync-pool, Description: Workload Identity Pool for DataSync

    • Click Continue

  3. Obtain the issuer URL for your AWS account. DataSync federates outbound using an account-specific issuer that hosts the OpenID Connect (OIDC) discovery endpoints. You register this issuer with your identity provider.

    1. Enable outbound web identity federation for your AWS account. This is a one-time, account-wide setting. If the feature is already enabled, this call returns an error that you can ignore.

      % aws iam enable-outbound-web-identity-federation
    2. Retrieve the issuer URL for your account:

      % aws iam get-outbound-web-identity-federation-info

      The result is similar to

      { "IssuerIdentifier": "https://a1755d5f-34cd-4f3c-9353-0ab65648e333.tokens.sts.global.api.aws", "JwtVendingEnabled": true }

      The IssuerIdentifier value is your issuer URL. The subject (sub) that your identity provider must trust is the ARN of the IAM role you created in the previous step, in the form arn:aws:iam::CUSTOMER_ACCOUNT:role/ROLE_NAME.

  4. Add an OIDC provider to the pool. Click Add Provider and select the OpenID Connect (OIDC) provider type, then enter the Provider details:

    • Provider name: e.g. aws-oidc-provider

    • Issuer (URL): the issuer URL obtained above

    • Audiences: select Allowed audiences and enter: //iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/providers/PROVIDER_NAME

    • Click Continue

  5. Under Attribute mapping, add:

    • google.subject = assertion.sub

    • attribute.aws_role = assertion.sub

    • Click Save

  6. Grant GCS bucket access:

    • Go to Cloud Storage → Buckets → your bucket → Permissions

    • Click Grant Access

    • In New principals, enter: principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/attribute.aws_role/arn:aws:iam::AWS_ACCOUNT_ID:role/ROLE_NAME

    • For Role, select a role with sufficient access rights, e.g. Storage Object Admin

    • Click Save

  7. Configure the DataSync location, e.g. using DataSync CLI:

    % aws datasync create-location-object-storage \ --server-hostname storage.googleapis.com \ --subdirectory prefix-within-the-bucket \ --bucket-name bucket-name \ --federated-identity 'AwsIamRole=arn:aws:iam::CUSTOMER_ACCOUNT:role/ROLE_NAME,ExternalIdentity={GoogleOidc={ProjectName=PROJECT_NAME,ProjectNumber=PROJECT_NUMBER,IdentityPoolName=POOL_NAME,IdentityProviderName=PROVIDER_NAME}}'

Step 2: Configure your network

Network configuration is required only when using a DataSync agent with your transfer. The network requirements for this migration depend on where you choose to deploy your agent.

If you want to host your DataSync agent in Google Cloud, configure your network to allow DataSync transfers through a public endpoint.

If you want to host your agent in AWS, you need a VPC with an interface endpoint. DataSync uses the VPC endpoint to facilitate the transfer.

To configure your network for a VPC endpoint
  1. If you don't have one, create a VPC in the same AWS Region as your S3 bucket.

  2. Create a private subnet for your VPC.

  3. Create a VPC service endpoint for DataSync.

  4. Configure your network to allow DataSync transfers through a VPC service endpoint.

    To do this, modify the security group that's associated with your VPC service endpoint.

Step 3: Create a DataSync agent (optional)

Most transfers require a DataSync agent. Transfers between Google Cloud Storage (GCS) and Amazon S3 using Enhanced mode do not require an agent. Use the agent that corresponds to your task mode. If you use an agent, it needs access to your GCS bucket.

In this scenario, the DataSync agent runs in your Google Cloud environment.

Before you begin: Install the Google Cloud CLI.

To create the agent for Google Cloud
  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, choose Agents, then choose Create agent.

  3. For Hypervisor, choose VMware ESXi, then choose Download the image.

    • The Enhanced mode agent downloads as an .ova image file.

    • The Basic mode agent downloads in a .zip file that contains the .ova image file.

  4. If you downloaded a Basic mode agent, open a terminal and unzip the image by running the following command:

    unzip AWS-DataSync-Agent-VMWare.zip
  5. Extract the contents of the agent's .ova file beginning with aws-datasync by running the following command:

    tar -xvf aws-datasync-2.0.1655755445.1-x86_64.xfs.gpt.ova
  6. Import the agent's .vmdk file into Google Cloud by running the following Google Cloud CLI command:

    gcloud compute images import aws-datasync-2-test \ --source-file INCOMPLETE-aws-datasync-2.0.1655755445.1-x86_64.xfs.gpt-disk1.vmdk \ --os centos-7
    Note

    Importing the .vmdk file might take up to two hours.

  7. Create and start a VM instance for the agent image that you just imported.

    The instance needs the following configurations for your agent. (For detailed instructions on how to create an instance, see the Google Cloud Compute Engine documentation.)

    • For the machine type, choose one of the following:

      • e2-standard-8 – For DataSync task executions working with up to 20 million objects.

      • e2-standard-16 – For DataSync task executions working with more than 20 million objects.

    • For the boot disk settings, go to the custom images section. Then choose the DataSync agent image that you just imported.

    • For the service account setting, choose your Google service account (the same account that you used in Step 1).

    • For the firewall setting, choose the option to allow HTTP (port 80) traffic.

      To activate your DataSync agent, port 80 must be open on the agent. The port doesn't need to be publicly accessible. Once activated, DataSync closes the port.

  8. After the VM instance is running, take note of its public IP address.

    You'll need this IP address to activate the agent.

  9. Go back to the DataSync console. On the Create agent screen where you downloaded the agent image, do the following to activate your agent:

    • For Endpoint type, choose the public service endpoints option (for example, Public service endpoints in US East Ohio).

    • For Activation key, choose Automatically get the activation key from your agent.

    • For Agent address, enter the public IP address of the agent VM instance that you just created.

    • Choose Get key.

  10. Give your agent a name, and then choose Create agent.

Your agent is online and ready to transfer data.

In this scenario, the agent runs as an Amazon EC2 instance in a VPC that's associated with your AWS account.

Before you begin: Set up the AWS Command Line Interface (AWS CLI).

To create the agent for your VPC
  1. Open a terminal. Make sure to configure your AWS CLI profile to use the account that's associated with your S3 bucket.

  2. Copy one of the following commands, depending on your task mode. Replace vpc-region with the AWS Region where your VPC resides (for example, us-east-1).

    Basic mode agents

    aws ssm get-parameter --name /aws/service/datasync/ami --region vpc-region

    Enhanced mode agents

    aws ssm get-parameter --name /aws/service/datasync/ami/v3 --region vpc-region
  3. Run the command. In the output, take note of the "Value" property.

    This value is the DataSync Amazon Machine Image (AMI) ID of the Region that you specified. For example, an AMI ID could look like ami-1234567890abcdef0.

  4. Copy the following URL. Again, replace vpc-region with the AWS Region where your VPC resides. Then, replace ami-id with the AMI ID that you noted in the previous step.

    https://console.aws.amazon.com/ec2/v2/home?region=vpc-region#LaunchInstanceWizard:ami=ami-id
  5. Paste the URL into a browser.

    The Amazon EC2 instance launch page in the AWS Management Console displays.

  6. For Instance type, choose one of the recommended Amazon EC2 instances for DataSync agents.

  7. For Key pair, choose an existing key pair, or create a new one.

  8. For Network settings, choose the VPC and subnet where you want to deploy the agent.

  9. Choose Launch instance.

  10. Once the Amazon EC2 instance is running, choose your VPC endpoint.

  11. Activate your agent.

Step 4: Create a DataSync source location for your Google Cloud Storage bucket

To set up a DataSync location for your Google Cloud Storage bucket, you need the access ID and secret for the HMAC key that you created in Step 1.

To create the DataSync source location
  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

  3. For Location type, choose Object storage.

  4. For Server, enter storage.googleapis.com.

  5. For Bucket name, enter the name of your Google Cloud Storage bucket.

  6. For Folder, enter an object prefix.

    DataSync only copies objects with this prefix.

  7. If your transfer requires an agent, choose Use agents, then choose the agent that you created in Step 3.

  8. Expand Additional settings. For Server protocol, choose HTTPS. For Server port, choose 443.

  9. Scroll down to the Authentication section. Make sure that the Requires credentials check box is selected, and then do the following:

    • For Access key, enter your HMAC key's access ID.

    • For Secret key, either enter your HMAC key's secret key directly, or specify an AWS Secrets Manager secret that contains the key. For more information, see Providing credentials for storage locations.

  10. Choose Create location.

Step 5: Create a DataSync destination location for your S3 bucket

You need a DataSync location for where you want your data to end up.

To create the DataSync destination location
  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

  3. Create a DataSync location for the S3 bucket.

    If you deployed the DataSync agent in your VPC, this tutorial assumes that the S3 bucket is in the same AWS Region as your VPC and DataSync agent.

Step 6: Create and start a DataSync task

With your source and destinations locations configured, you can start moving your data into AWS.

To create and start the DataSync task
  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, expand Data transfer, then choose Tasks, and then choose Create task.

  3. On the Configure source location page, do the following:

    1. Choose Choose an existing location.

    2. Choose the source location that you created in Step 4, then choose Next.

  4. On the Configure destination location page, do the following:

    1. Choose Choose an existing location.

    2. Choose the destination location that you created in Step 5, then choose Next.

  5. On the Configure settings page, do the following:

    1. Under Data transfer configuration, expand Additional settings and clear the Copy object tags check box.

      Important

      Because the Google Cloud Storage XML API does not support reading or writing object tags, your DataSync task might fail if you try to copy object tags.

    2. Configure any other task settings that you want, and then choose Next.

  6. On the Review page, review your settings, and then choose Create task.

  7. On the task's details page, choose Start, and then choose one of the following:

    • To run the task without modification, choose Start with defaults.

    • To modify the task before running it, choose Start with overriding options.

When your task finishes, you'll see the objects from your Google Cloud Storage bucket in your S3 bucket.