Working with Locations

In this section, you can find information about how to create and configure locations.

A location is an endpoint of a task. AWS DataSync supports Network File System (NFS), Amazon EFS, and Amazon S3 as location types. For NFS and Amazon EFS, the location is the export path where you want to mount your file system. For Amazon S3, the location is the prefix path that you want to mount and use as the root of the sync.

Topics

Creating a Location for NFS
Creating a Location for Amazon EFS
Creating a Location for Amazon S3
How DataSync Stores Files in Locations
Deleting a Location

AWS DataSync supports the following location combinations.

Source (From)	Destination (To)
On-premises NFS file system	Amazon EFS file system
On-premises NFS file system	Amazon S3
Amazon EFS	On-premises NFS file system
Amazon S3	On-premises NFS file system

Creating a Location for NFS

DataSync supports the NFS v3, NFS v4.0, and NFS v4.1 protocols.

To create an NFS location

Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.
In the navigation pane, choose Locations. The locations that you previously created appear in the list of locations.
On the Locations page, choose Create location.
For Type, choose NFS. You configure this location as a source or destination later.
For Agent, choose the agent you want to use. If you have previously created agents, the agents appear in the list. The agent connects to your on-premises NFS server and makes it easier to securely transfer data between the on-premises location and AWS.
For NFS server, provide the DNS name or IP address of the NFS server.

DataSync automatically chooses the NFS version that it uses to read from an NFS location. If you need to force DataSync to use a specific NFS version, use the optional Version parameter for the NfsMountOptions API operation.
For Mount path, enter the mount path for your NFS location.
(Optional) For Key and Value, enter a key and value to tag your NFS location. A tag is a key-value pair that helps you manage, filter, and search for your locations.
When you are done, choose Create location.

For detailed information about these NFS location settings, see NFS Location Settings.

NFS Location Settings

Following, you can find descriptions for the configuration settings for NFS locations in DataSync.

Agent

An agent is a VM that is deployed in your on-premises environment to connect to your on-premises location. An agent makes it easier to securely transfer data between the on-premises location and AWS. You can use an agent for more than one location.

If a task is using multiple agents, all the agents need to have the status Available for the task to run. If you use multiple agents for a source location, the status of all the agents must be Available for the task to run. Agents are automatically updated by AWS on a regular basis, using a mechanism that doesn't interrupt your tasks.

NFS server

The name of the NFS server, the IP address or DNS name of the NFS server. An agent that is installed on-premises uses this name to mount the NFS server in a network.

Mount path

The mount path for your NFS file system. This path must be a path that's exported by the NFS server, or a subdirectory of an exported path. This path should be such that it can be mounted by other NFS clients in your network. For information about how to resolve mount path issues, see Your Task Status is Unavailable and Status Indicates a Mount Error.

To transfer all the data in the folder you specified, DataSync needs to have permissions to read all the data. To ensure this, either configure the NFS export with no_root_squash, or ensure that the permissions for all of the files you want DataSync to allow read access for all users. Doing either enables the agent to read the files. For the agent to access directories, you must additionally enable all execute access.

Tag

A tag is a key-value pair that helps you manage, filter, and search for your location. Adding a tag is optional. We recommend using tags for naming your resources.

Forcing DataSync to Use a Specific NFS Version to Mount Your NFS Share

DataSync automatically chooses the NFS version that it uses when reading from an NFS location. If you need to force DataSync to use a specific NFS version, you can use the optional Version parameter for the CreateLocationNfs API operation. If you don't specify an NFS version, DataSync chooses one through negotiation with the NFS server.

The following CLI commands create an NFS source location and force DataSync to use NFS version 4.0.


$ aws datasync create-location-nfs --server-hostname
                            your-server-address --on-prem-config
                            AgentArns=your-agent-arns
                            --subdirectory nfs-export-path
                            --mount-option Version=NFS4_0

Creating a Location for Amazon EFS

A location for Amazon EFS is an endpoint for an Amazon EFS file system. If you don't have an Amazon EFS file system in the current AWS Region, create one. For information about how to create an Amazon EFS file system, see Getting Started with Amazon Elastic File System in the Amazon Elastic File System User Guide.

Note

DataSync currently doesn't support transferring files to Amazon EFS volumes that are in dedicated tenancy VPCs. For information about dedicated tenancy VPCs, see Creating a VPC with an Instance Tenancy of Dedicated in the Amazon EC2 User Guide for Linux Instances.

To create an EFS location

Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.
In the navigation pane, choose Locations. The locations that you previously created appear in the list of locations.
On the Create location page, choose EFS for Type.
For File system, choose the EFS file system that you want to use as an endpoint. You configure this location as a source or destination later.
For Mount path, enter the mount path for your EFS file system. The path can include a subdirectory. If so, this is a subdirectory in the EFS file system that is used to read data from the EFS location or write data to the EFS destination. By default, DataSync uses the root directory.
For Subnet and Security Group, the DataSync console automatically chooses a subnet that includes a mount target for your Amazon EFS file system and this subnet’s default security group. We recommend using these default settings.
Note

DataSync uses the security group specified in this step to connect to your Amazon EFS file system. If the security group is configured to disallow connections from within itself, you have two options. One is to change this configuration to allow the security group to communicate within itself. The other is to choose a different subnet and security group, such that the following is true:
- The selected subnet contains a mount target for your EFS file system.
- The selected security group can communicate with that mount target's security group.
For detailed information about security groups and mount targets, see Security Groups for Amazon EC2 Instances and Mount Targets in the Amazon Elastic File System User Guide.
(Optional) Provide values for the Key and Value fields to tag the EFS file system. A tag is a key-value pair that helps you manage, filter, and search for your locations. We recommend using tags for naming your resources.
When you are done, choose Create location. The location that you just created appears in the list of locations.

Considerations When Creating a Location for Amazon EFS

Be sure to consider the following when creating a location for Amazon EFS:

When you create an Amazon EFS file system in Bursting Throughput mode, you get an allocation of 2.1 TB worth of burst credits. All Amazon EFS file systems are able to burst up to 100 MB/s of throughput when using Bursting Throughput mode. File systems that are larger than 1 TiB can burst to twice their baseline throughput. Note that DataSync consumes file system burst credits. This can have an impact on the performance of your applications. When using DataSync with a file system that has an active workload, consider using EFS Provisioned Throughput.
Amazon EFS file systems that are in General Purpose performance mode have a limit of 7,000 file system operations per second. This limit can impact the maximum throughput DataSync can achieve when copying files.

For more information, see Amazon EFS Performance in the Amazon Elastic File System User Guide.

Creating a Location for Amazon S3

A location for Amazon S3 is an endpoint for the Amazon S3 bucket that DataSync uses as a source or destination.

Topics

Amazon S3 Location Settings
Manually Configuring an IAM Role to Access Your S3 Bucket

To create an S3 location

Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.
In the navigation pane, choose Locations. The locations that you previously created appear in the list of locations.
On the Create location page, choose Create location.
For Type, choose Amazon S3 bucket.
For S3 Bucket, choose the Amazon S3 bucket that you want to use as an endpoint. You configure this location as a source or destination later.
For Folder, provide the name of a folder in S3. This is the S3 folder that DataSync uses, either to read data from for an S3 source location or write data to for an S3 destination.
For IAM role, choose Autogenerate for DataSync to automatically create a role with the required permissions.

If DataSync has previously created such a role for this S3 bucket, that role is chosen as the default in the list. You can also create your own role and choose it from the list. For instructions on how to create an IAM role manually, see Manually Configuring an IAM Role to Access Your S3 Bucket .
(Optional) For Key and Value, provide values to tag your S3 location. A tag is a key-value pair that helps you manage, filter, and search for your locations.
When you are done, choose Create location. The location that you just created appears in the list of locations.

Note

In addition to the IAM policies that grant DataSync permissions, we recommend creating a multipart upload bucket policy for your S3 buckets to help you control your storage cost. For more information, see the blog post S3 Lifecycle Management Update – Support for Multipart Uploads and Delete Markers.

Amazon S3 Location Settings

If the location you want to use as a source or a destination is an Amazon S3 bucket, you configure the following settings.

S3 bucket

The Amazon S3 bucket that you want to use as a source or destination location.

IAM role

The AWS Identity and Access Management (IAM) role that has permissions to access the S3 bucket.

For AWS DataSync to access a destination S3 bucket, it requires access to your Amazon S3 bucket. To obtain this access, DataSync assumes the IAM role that you provide. The role requires an IAM policy and a security token service trust (STS) relationship. The policy determines which actions the role can perform. DataSync can create the role on your behalf. For instructions, see Creating a Location for Amazon S3 . You can also create the role manually and choose it from the list in the console. For instructions, see Manually Configuring an IAM Role to Access Your S3 Bucket .

Tag

A key-value pair that identifies the S3 location. By default, the DataSync console prepopulates a name value with the task or location name.

Manually Configuring an IAM Role to Access Your S3 Bucket

When you use the DataSync Management Console to create an Amazon S3 location, DataSync automatically creates an IAM role that has the required permissions for you. If you want to create the IAM role manually, use the following procedure.

To manually configure an IAM role to access your S3 bucket

Open the IAM Management Console.
From the navigation pane, choose Roles, and then choose Create role to open the Create role page.
In the Select type of trusted entity section, make sure that AWS service is selected.

Under Choose the service that will use this role, choose DataSync, or manually configure it as shown in the following example.


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "datasync.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Choose Next: Permissions.
Choose AmazonS3FullAccess. You can also manually configure a more restricted policy. For an example of such a policy, see Amazon S3 Location Settings.
(Optional) Choose Next: Tags to create tags for the role.
Choose Next: Review, choose the role name, and then choose Create role.
Open the AWS DataSync Management Console.
Choose the refresh button on the right side of the IAM role list, and then choose the role that you just created.

How DataSync Stores Files in Locations

How DataSync transfers and stores files depends on the file type and the location type. The following table shows how files are stored in each of the location types.

File Type	NFS	Amazon EFS	Amazon S3
Regular	Natively	Natively	Object
Symbolic link	Natively	Natively	Link content stored in object
Directory	Natively	Natively	Empty object
Block device	Natively	Not supported	Empty object
Character device	Natively	Not supported	Empty object
Named pipe	Ignored	Ignored	Ignored
Socket	Ignored	Ignored	Ignored

The following table shows how metadata is transferred and stored in each location type.

POSIX Metadata	NFS	Amazon EFS	Amazon S3
File type	inode mode field	inode mode field	Object user-defined metadata file permissions The exception is directories that are indicated by a trailing / in the object n.
Permissions	inode mode field	inode mode field	Object user-defined metadata file permissions
User ID	inode uid field	inode uid field	Object user-defined metadata file owner
Group ID	inode uid field	inode uid field	Object user-defined metadata file group
Modification time	inode Mtime field	inode Mtime field	Object user-defined metadata file Mtime
Access time	inode Atime field	inode Atime field	Object user-defined metadata file Atime
Sticky/Setuid/Setgid	inode mode field	inode mode field	Object user-defined metadata file permissions
Device major/minor	inode device id field	Not supported	Object user-defined metadata file device
Hard links	inode link cound field	inode link cound field	Object user-defined metadata file inode

When transferring files from an Amazon S3 location to an Amazon EFS or NFS location, objects that don't have POSIX metadata assume the default UID, GID, folder, and file permissions. The following table shows the default permissions that DataSync assumes.

Permission	Value
UID	65534
GID	65534
Folder Permission	0755
File Permission	0755

Deleting a Location

Use the following procedure to delete any type of location.

To delete a location

Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.
On the navigation pane, choose Locations.
On the Locations page, choose the location that you want to delete.
Choose Delete, note the location ID that appears, enter delete in the text box, and choose Delete.

Warning Javascript is disabled or is unavailable in your browser.

To use the AWS Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

« Previous Next »