Step 2: Create Locations - AWS DataSync

Step 2: Create Locations

Each DataSync task is made up of a pair of locations between which data is transferred. The source location defines the storage system or service that you want to read data from. The destination location defines the storage system or service that you want to write data to.

For a list of all DataSync supported source and destination endpoints, see Working with Locations.

Create an NFS Location

Use the following procedure to create an NFS location using the AWS CLI. An NFS location defines a file system on an NFS server that can be read from or written to. You can also create an NFS location using the AWS Management Console. For more information, see Creating a Location for NFS.

Note

If you are using an NFS location on an AWS Snowcone device, see NFS Server on AWS Snowcone for more information about transferring data to or from that device.

To create an NFS location using the CLI

  • Use the following command to create an NFS source location.

    $ aws datasync create-location-nfs --server-hostname server-address --on-prem-config AgentArns=agent-arns --subdirectory nfs-export-path

    For the preceding command, the following applies:

    • The path that you provide for the --subdirectory parameter should be a path that's exported by the NFS server, or a subdirectory. Other NFS clients in your network should be able to mount this path. To see all the paths exported by your NFS server, run the command showmount -e nfs-server-address from an NFS client with access to your server. You can specify any directory that appears in the results, and any subdirectory of that directory.

    • To transfer all the data in the folder that you specified, DataSync needs permissions to read all the data. To give DataSync permissions, you can do one of two things. You can configure the NFS export with no_root_squash. Or, for the all files that you want DataSync to access, you can make sure that the permissions allow read access for all users. Doing either enables the agent to read the files. For the agent to access directories, you must additionally give all users execute access.

    • Make sure that the NFS export is accessible without Kerberos authentication.

    DataSync automatically chooses the NFS version that it uses to read from an NFS location. To specify an NFS version, use the optional Version parameter in the NfsMountOptions API operation.

This command returns the Amazon Resource Name (ARN) of the NFS location, similar to the ARN shown following.

{ "LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0f01451b140b2af49" }

To make sure that the directory can be mounted, you can connect to any computer that has the same network configuration as your agent and run the following command.

mount -t nfs -o nfsvers=<nfs-server-version <nfs-server-address:<nfs-export-path <test-folder

The following is an example of the command.

mount -t nfs -o nfsvers=3 198.51.100.123:/path_for_sync_to_read_from /temp_folder_to_test_mount_on_local_machine

Create an SMB Location

Use the following procedure to create an SMB location using the AWS CLI. An SMB location defines a file system on an SMB server that can be read from or written to. You can also create an SMB location using the console. For more information, see Creating a Location for SMB.

To create an SMB location using the CLI

  • Use the following command to create an SMB source location.

    $ aws datasync create-location-smb --server-hostname smb-server-address --user user-name --domain domain-of-the-smb-server --password user's-password AgentArns=agent-arns --subdirectory smb-export-path

    The path that you provide for the --subdirectory parameter should be a path that's exported by the SMB server, or a subdirectory. Specify the path using forward slashes, for example /path/to/folder. Other SMB clients in your network should be able to access this path.

    DataSync automatically chooses the SMB version that it uses to read from an SMB location. To specify an SMB version, use the optional Version parameter in the SmbMountOptions API operation.

This command returns the Amazon Resource Name (ARN) of the SMB location, similar to the ARN shown following.

{ "LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0f01451b140b2af49" }

Create an Object Storage Location

Use the following procedure to create a self-managed object storage location using the AWS CLI. An object storage location is the endpoint for an Amazon S3 API compatible object storage server. An object storage location defines an object storage server that can be read from or written to. You can also create an object storage location using the AWS Management Console.

For more information about object storage locations, including compatibility requirements, see Creating a Location for Object Storage.

To create a self-managed object storage location using the CLI

  • Use the following command to create a self-managed object storage location.

    $ aws datasync create-location-object-storage --server-hostname object-storage-server.example.com --bucket-name myBucket --agent-arns arn:aws:datasync:us-east-1:123456789012:agent/agent-01234567890deadfb

    The following parameters are required in the create-location-object-storage command.

    • server-hostname: The DNS name or IP address of the self-managed object storage server.

    • bucket-name: The name that identifies the bucket on the self-managed object storage server at the location.

    • agent-arns: The ARNs of the agents to use for the self-managed object storage location.

    If your object storage requires a user name and password to authenticate, use the --access-key and --secret-key parameters to provide the user name and password, respectively.

The preceding command returns a location ARN similar to the following.

{ "arn:aws:datasync:us-east-1:123456789012:location/loc-01234567890deadfb" }

Create an Amazon EFS Location

Use the following procedure to create an EFS location using the AWS CLI. An EFS location is the endpoint for an Amazon EFS file system, which defines an EFS file system that can be read from or written to. You can also create an EFS location using the console. For more information, see Creating a Location for Amazon EFS.

To create an Amazon EFS location using the CLI

  1. If you don't have an Amazon EFS file system, create one. For information about how to create an EFS file system, see Getting Started with Amazon Elastic File System in the Amazon Elastic File System User Guide.

  2. Identify a subnet that has at least one mount target for that file system. You can see all the mount targets and the subnets associated with an EFS file system by using the describe-mount-targets command.

    $ aws --region aws-region efs describe-mount-targets --file-system-id file-system-id
    Note

    The AWS Region that you specify is the one where your target S3 bucket or EFS file system is located.

    This command returns information about the target similar to the information shown following.

    { "MountTargets": [ { "OwnerId": "111222333444", "MountTargetId": "fsmt-22334a10", "FileSystemId": "fs-123456ab", "SubnetId": "subnet-f12a0e34", "LifeCycleState": "available", "IpAddress": "11.222.0.123", "NetworkInterfaceId": "eni-1234a044" } ] }
  3. Specify an Amazon EC2 security group that can be used to access the mount target. You can run the following command to find out the security group of the mount target.

    $ aws --region aws-region efs describe-mount-target-security-groups --mount-target-id mount-target-id

    The security group that you provide needs to be able to communicate with the security group on the mount target in the subnet specified.

    The relationship between security group M on the mount target and security group S, which you provide for DataSync to use at this stage, is as follows:

    • Security group M, which you associate with the mount target, must allow inbound access for the TCP protocol on the NFS port (2049) from security group S.

      You can enable an inbound connection either by its IP address (CIDR range) or its security group.

    • Security group S, which you provide to DataSync to access EFS, should have a rule that enables outbound connections to the NFS port. It enables outbound connections on one of the file system's mount targets.

      You can enable outbound connections either by IP address (CIDR range) or security group.

      For information about security groups and mount targets, see Security Groups for Amazon EC2 Instances and Mount Targets in the Amazon Elastic File System User Guide.

  4. Create the EFS location. To create the EFS location, you need the ARNs for your Amazon EC2 subnet, EC2 security group, and an EFS file system. Because the DataSync API accepts fully qualified ARNs, you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the AWS General Reference.

    Use the following command to create an EFS location.

    $ aws datasync create-location-efs --subdirectory /path/to/your/subdirectory --efs-filesystem-arn 'arn:aws:elasticfilesystem:region:account-id:file-system/filesystem-id' --ec2-config SecurityGroupArns='arn:aws:ec2:region:account-id:security-group/security-group-id',SubnetArn='arn:aws:ec2:region:account-id:subnet/subnet-id'
Note

The AWS Region that you specify is the one where your target S3 bucket or EFS file system is located.

The command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Create an Amazon FSx for Windows File Server Location

Use the following procedure to create an Amazon FSx for Windows File Server location using the AWS CLI. An Amazon FSx location is the endpoint for an Amazon FSx for Windows File Server. This endpoint defines the Amazon FSx file share that you can read from or write to.

You can also create an Amazon FSx location using the console. For more information, see Creating a Location for Amazon FSx for Windows File Server.

To create an Amazon FSx for Windows File Server location using the CLI

  • Use the following command to create an Amazon FSx location.

    $ aws datasync create-location-fsx-windows \ --fsx-filesystem-arn arn:aws:fsx:region:account-id:file-system/filesystem-id \ --security-group-arns arn:aws:ec2:region:account-id:security-group/group-id \ --user smb-user --password password

    In the create-location-fsx-windows command, specify the following:

    • The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.

      The DataSync API accepts fully qualified ARNs, and you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the AWS General Reference.

    • The ARN of an Amazon EC2 security group that can be applied to the Elastic Network Interfaces of the file system's preferred subnet. For more information, see Creating a VPC with an Instance Tenancy of Dedicated in the Amazon EC2 User Guide.

    • The AWS Region. The Region that you specify is the one where your target Amazon FSx file system is located.

The preceding command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Create an Amazon S3 Location

Use the following procedure to create an Amazon S3 location using the AWS CLI. An Amazon S3 location requires an Amazon S3 bucket that can be read from or written to. To create an Amazon S3 bucket, see Creating a bucket in the Amazon S3 Console User Guide.

For DataSync to access an Amazon S3 bucket, DataSync needs an AWS Identity and Access Management (IAM) role that has the required permissions. With the following procedure, you create the IAM role, the required IAM policies, and the S3 location using the AWS CLI.

You can also create an S3 location using the console. For more information, see Creating a Location for Amazon S3.

To create an S3 location using the CLI

  1. Create an IAM trust policy that allows DataSync to assume the IAM role required to access your S3 bucket.

    The following is an example of a trust policy.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
  2. Create a temporary file for the IAM policy, as shown in the following example.

    $ ROLE_FILE=$(mktemp -t sync.iam.role.XXXXXX.json) $ IAM_ROLE_NAME='YourBucketAccessRole' $ cat<<EOF> ${ROLE_FILE} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF
  3. Create an IAM role and attach the IAM policy to it.

    The following command creates an IAM role and attaches the policy to it.

    $ aws iam create-role --role-name ${IAM_ROLE_NAME} --assume-role-policy-document file://${ROLE_FILE} { "Role": { "Path": "/", "RoleName": "YourBucketAccessRole", "RoleId": "role-id", "Arn": "arn:aws:iam::account-id:role/YourBucketAccessRole", "CreateDate": "2018-07-27T02:49:23.117Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } }
  4. Allow the IAM role that you created to write to your S3 bucket.

    Attach an IAM policy that has sufficient permissions to access your S3 bucket to the IAM role. The following example shows the minimum permissions needed for DataSync to read and write to an S3 bucket in an AWS Region.

    { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketLocation", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:HeadBucket" ], "Effect": "Allow", "Resource": "YourS3BucketArn" }, { "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:ListMultipartUploadParts", "s3:PutObjectTagging", "s3:GetObjectTagging", "s3:PutObject" ], "Effect": "Allow", "Resource": "YourS3BucketArn/*" } ] }

    To attach the policy to your IAM role, run the following command.

    $ aws iam attach-role-policy --role-name role-name --policy-arn 'arn:aws:iam::aws:policy/YourPolicyName'

    For Amazon S3 buckets on AWS Outposts, use the following policy.

    { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3-outposts:ListBucket", "s3-outposts:ListBucketMultipartUploads" ], "Effect": "Allow", "Resource": [ "s3OutpostsBucketArn", "s3OutpostsAccessPointArn" ], "Condition": { "StringLike": { "s3-outposts:DataAccessPointArn": "s3OutpostsAccessPointArn" } } }, { "Action": [ "s3-outposts:AbortMultipartUpload", "s3-outposts:DeleteObject", "s3-outposts:GetObject", "s3-outposts:ListMultipartUploadParts", "s3-outposts:PutObjectTagging", "s3-outposts:GetObjectTagging", "s3-outposts:PutObject" ], "Effect": "Allow", "Resource": [ "s3OutpostsBucketArn/*", "s3OutpostsAccessPointArn" ], "Condition": { "StringLike": { "s3-outposts:DataAccessPointArn": "s3OutpostsAccessPointArn" } } }, { "Effect": "Allow", "Action": [ "s3-outposts:GetAccessPoint" ], "Resource": "s3OutpostsAccessPointArn" } ] }
  5. Create the S3 location.

    Use the following command to create your Amazon S3 location.

    $ aws datasync create-location-s3 --s3-bucket-arn 'arn:aws:s3:::bucket' --s3-storage-class 'your-S3-storage-class' --s3-config 'BucketAccessRoleArn=arn:aws:iam::account-id:role/role-allowing-DS-operations' --subdirectory /your-folder

    The command returns a location ARN similar to the one shown following.

    { "LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d" }

    The location type information is encoded in the LocationUri. In this example, the s3:// prefix in LocationUri shows the location's type.

    If your Amazon S3 bucket is located on an AWS Outpost, you must deploy an Amazon EC2 agent on your Outpost. The agent must be in a VPC that is allowed to access the access point specified in the command. The agent also must be activated in the Outpost's parent Region, and be able to route to the Amazon S3 on Outposts endpoints for the bucket. For more information about launching a DataSync agent on AWS Outposts, see Deploy Your DataSync Agent on AWS Outposts.

    Use the following command to create an Amazon S3 location on your Outpost.

    $ aws datasync create-location-s3 --s3-bucket-arn access-point-arn --s3-config BucketAccessRoleArn=arn:aws:iam::account-id:role/role-allowing-DS-operations --agent-arns datasync-agent-arn-in-the-vpc-which-can-access-your-s3-acces-point
Note
  • Changes to object data or metadata are equivalent to deleting an object and creating a new one to replace it. This results in additional charges in the following scenarios:

    • When using object versioning: Changes to object data or metadata create a new version of the object.

    • When using storage classes that can incur additional charges for overwriting, deleting, or retrieving, objects: Changes to object data or metadata result in such charges. For more information, see Considerations When Working with Amazon S3 Storage Classes in DataSync.

  • When you use object versioning, a single DataSync task execution might create more than one version of an Amazon S3 object.

  • In addition to the IAM policies that grant DataSync permissions, we recommend creating a multipart upload bucket policy for your S3 buckets. Doing this can help you control your storage costs. For more information, see the blog post S3 Lifecycle Management Update – Support for Multipart Uploads and Delete Markers.