Step 2: Create Locations - AWS DataSync

Step 2: Create Locations

Each DataSync task is made up of a pair of locations between which data is transferred. The source location defines the storage system or service that you want to read data from. The destination location defines the storage system or service that you want to write data to.

For a list of all DataSync supported source and destination endpoints, see Working with Locations.

Create an NFS Location

Use the following procedure to create an NFS location using the AWS CLI. An NFS location defines a file system on an NFS server that can be read from or written to. You can also create an NFS location using the AWS Management Console. For more information, see Creating a Location for NFS.

To create an NFS location using the CLI

  • Use the following command to create an NFS source location.

    $ aws datasync create-location-nfs --server-hostname server-address --on-prem-config AgentArns=agent-arns --subdirectory nfs-export-path

    For the preceding command, the following applies:

    • The path that you provide for the --subdirectory parameter should be a path that's exported by the NFS server, or a subdirectory. Other NFS clients in your network should be able to mount this path. To see all the paths exported by your NFS server, run the command showmount -e nfs-server-address from an NFS client with access to your server. You can specify any directory that appears in the results, and any subdirectory of that directory.

    • To transfer all the data in the folder that you specified, DataSync needs permissions to read all the data. To give DataSync permissions, you can do one of two things. You can configure the NFS export with no_root_squash. Or, for the all files that you want DataSync to access, you can make sure that the permissions allow read access for all users. Doing either enables the agent to read the files. For the agent to access directories, you must additionally give all users execute access.

    • Make sure that the NFS export is accessible without Kerberos authentication.

    DataSync automatically chooses the NFS version that it uses to read from an NFS location. To specify an NFS version, use the optional Version parameter in the NfsMountOptions API operation.

This command returns the Amazon Resource Name (ARN) of the NFS location, similar to the ARN shown following.

{ "LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0f01451b140b2af49" }

To make sure that the directory can be mounted, you can connect to any computer that has the same network configuration as your agent and run the following command.

mount -t nfs -o nfsvers=<nfs-server-version <nfs-server-address:<nfs-export-path <test-folder

The following is an example of the command.

mount -t nfs -o nfsvers=3 198.51.100.123:/path_for_sync_to_read_from /temp_folder_to_test_mount_on_local_machine

Create an SMB Location

Use the following procedure to create an SMB location using the AWS CLI. An SMB location defines a file system on an SMB server that can be read from or written to. You can also create an SMB location using the console. For more information, see Creating a Location for SMB.

To create an SMB location using the CLI

  • Use the following command to create an SMB source location.

    $ aws datasync create-location-smb --server-hostname smb-server-address --user user-name --domain domain-of-the-smb-server --password user's-password AgentArns=agent-arns --subdirectory smb-export-path

    The path that you provide for the --subdirectory parameter should be a path that's exported by the SMB server, or a subdirectory. Specify the path using forward slashes, for example /path/to/folder. Other SMB clients in your network should be able to access this path.

    DataSync automatically chooses the SMB version that it uses to read from an SMB location. To specify an SMB version, use the optional Version parameter in the SmbMountOptions API operation.

This command returns the Amazon Resource Name (ARN) of the SMB location, similar to the ARN shown following.

{ "LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0f01451b140b2af49" }

Create an Amazon EFS Location

Use the following procedure to create an EFS location using the AWS CLI. An EFS location is the endpoint for an Amazon EFS file system, which defines an EFS file system that can be read from or written to. You can also create an EFS location using the console. For more information, see Creating a Location for Amazon EFS.

To create an Amazon EFS location using the CLI

  1. If you don't have an Amazon EFS file system, create one. For information about how to create an EFS file system, see Getting Started with Amazon Elastic File System in the Amazon Elastic File System User Guide.

  2. Identify a subnet that has at least one mount target for that file system. You can see all the mount targets and the subnets associated with an EFS file system by using the describe-mount-targets command.

    $ aws --region aws-region efs describe-mount-targets --file-system-id file-system-id
    Note

    The AWS Region that you specify is the one where your target S3 bucket or EFS file system is located.

    This command returns information about the target similar to the information shown following.

    { "MountTargets": [ { "OwnerId": "111222333444", "MountTargetId": "fsmt-22334a10", "FileSystemId": "fs-123456ab", "SubnetId": "subnet-f12a0e34", "LifeCycleState": "available", "IpAddress": "11.222.0.123", "NetworkInterfaceId": "eni-1234a044" } ] }
  3. Specify an Amazon EC2 security group that can be used to access the mount target. You can run the following command to find out the security group of the mount target.

    $ aws --region aws-region efs describe-mount-target-security-groups --mount-target-id mount-target-id

    The security group that you provide needs to be able to communicate with the security group on the mount target in the subnet specified.

    The relationship between security group M on the mount target and security group S, which you provide for DataSync to use at this stage, is as follows:

    • Security group M, which you associate with the mount target, must allow inbound access for the TCP protocol on the NFS port (2049) from security group S.

      You can enable an inbound connection either by its IP address (CIDR range) or its security group.

    • Security group S, which you provide to DataSync to access EFS, should have a rule that enables outbound connections to the NFS port. It enables outbound connections on one of the file system's mount targets.

      You can enable outbound connections either by IP address (CIDR range) or security group.

      For information about security groups and mount targets, see Security Groups for Amazon EC2 Instances and Mount Targets in the Amazon Elastic File System User Guide.

  4. Create the EFS location. To create the EFS location, you need the ARNs for your Amazon EC2 subnet, EC2 security group, and an EFS file system. Because the DataSync API accepts fully qualified ARNs, you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the AWS General Reference.

    Use the following command to create an EFS location.

    $ aws datasync create-location-efs --subdirectory /path/to/your/subdirectory --efs-filesystem-arn 'arn:aws:elasticfilesystem:region:account-id:file-system/filesystem-id' --ec2-config SecurityGroupArns='arn:aws:ec2:region:account-id:security-group/security-group-id',SubnetArn='arn:aws:ec2:region:account-id:subnet/subnet-id'
Note

The AWS Region that you specify is the one where your target S3 bucket or EFS file system is located.

The command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Create an Amazon FSx for Windows File Server Location

Use the following procedure to create an Amazon FSx for Windows File Server location using the AWS CLI. An Amazon FSx location is the endpoint for an Amazon FSx for Windows File Server. This endpoint defines the Amazon FSx file share that you can read from or write to.

You can also create an Amazon FSx location using the console. For more information, see Creating a Location for Amazon FSx for Windows File Server

To create an Amazon FSx for Windows File Server location using the CLI

  • Use the following command to create an Amazon FSx location.

    $ aws datasync create-location-fsx-windows \ --fsx-filesystem-arn arn:aws:fsx:region:account-id:file-system/filesystem-id \ --security-group-arns arn:aws:ec2:region:account-id:security-group/group-id \ --user smb-user --password password

    In the create-location-fsx-windows command, specify the following:

    • The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.

      The DataSync API accepts fully qualified ARNs, and you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the AWS General Reference.

    • The ARN of an Amazon EC2 security group that can be applied to the Elastic Network Interfaces of the file system's preferred subnet. For more information, see Creating a VPC with an Instance Tenancy of Dedicated in the Amazon EC2 User Guide.

    • The AWS Region. The Region that you specify is the one where your target Amazon FSx file system is located.

The preceding command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Create an Amazon S3 Location

Use the following procedure to create an Amazon S3 location using the AWS CLI. An Amazon S3 location requires an Amazon S3 bucket that can be read from or written to. To create an Amazon S3 bucket, see Create a Bucket in the Amazon S3 Console User Guide.

For DataSync to access a destination Amazon S3 bucket, DataSync needs an AWS Identity and Access Management (IAM) role that has the required permissions. With the following procedure, you create the IAM role, required IAM policies, and the S3 location using the AWS CLI.

You can also create an S3 location using the console. For more information, see Creating a Location for Amazon S3.

To create an S3 location using the CLI

  1. Create an IAM trust policy that allows DataSync to assume the IAM role required to access your S3 bucket.

    The following is an example of a trust policy.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
  2. Create a temporary file for the IAM policy, as shown in the following example.

    $ ROLE_FILE=$(mktemp -t sync.iam.role.XXXXXX.json) $ IAM_ROLE_NAME='YourBucketAccessRole' $ cat<<EOF> ${ROLE_FILE} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF
  3. Create an IAM role and attach the IAM policy to it.

    The following command creates an IAM role and attaches the policy to it.

    $ aws iam create-role --role-name ${IAM_ROLE_NAME} --assume-role-policy-document file://${ROLE_FILE} { "Role": { "Path": "/", "RoleName": "YourBucketAccessRole", "RoleId": "role-id", "Arn": "arn:aws:iam::account-id:role/YourBucketAccessRole", "CreateDate": "2018-07-27T02:49:23.117Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } }
  4. Allow the IAM role that you created to write to your S3 bucket.

    Attach an IAM policy that has sufficient permissions to access your S3 bucket to the IAM role (for example, AmazonS3FullAccess). You can also create a policy that is more restrictive. If you do, the minimal permissions needed for DataSync to read and write to an S3 location are shown the following example.

    { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketLocation", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:HeadBucket" ], "Effect": "Allow", "Resource": "arn:aws:s3:::YourBucket" }, { "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Effect": "Allow", "Resource": "arn:aws:s3:::YourBucket/*" } ] }

    To attach the policy to your IAM role, run the following command.

    $ aws iam attach-role-policy --role-name role-name --policy-arn 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
  5. Create the S3 location.

    Use the following commands to create your Amazon S3 location.

    $ aws datasync create-location-s3 --s3-bucket-arn 'arn:aws:s3:::bucket' --s3-storage-class 'your-S3-storage-class' --s3-config 'BucketAccessRoleArn=arn:aws:iam::account-id:role/role-name' subdirectory /your-folder

    The commands return a location ARN similar to the one shown following.

    { "LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d" }

You can see information about your S3 location that you just created by using the describelocation-s3 command.

The location type information is encoded in the LocationUri of every location description, regardless of the location type. In the example preceding, the s3:// prefix in LocationUri shows the location’s type.

Note

If versioning is enabled for S3, and you configure DataSync to copy file metadata, DataSync creates a new object every time that the corresponding file’s metadata is updated.