Configuring AWS DataSync transfers with Amazon EFS - AWS DataSync

Configuring AWS DataSync transfers with Amazon EFS

To transfer data to or from your Amazon EFS file system, you must create an AWS DataSync transfer location. DataSync can use this location as a source or destination for transferring data.

Providing DataSync access to Amazon EFS file systems

DataSync mounts your Amazon EFS file system as the root user from your virtual private cloud (VPC) using network interfaces.

When creating your location, you specify the subnet and security groups that DataSync uses to connect to one of your Amazon EFS file system's mount targets or access points using Network File System (NFS) port 2049.

Note

If DataSync accesses a destination file system through an access point that enforces user identity, the POSIX user and group IDs for your source data aren't preserved if you configure your DataSync task to copy ownership. Instead, the transferred files and folders are set to the access point's user and group IDs. When this happens, task verification fails because DataSync detects a mismatch between metadata in the source and destination locations.

DataSync can also mount Amazon EFS file systems configured for restricted access. For example, you can specify an AWS Identity and Access Management (IAM) role that gives DataSync the necessary level of permission to connect to your file system. For more information, see Using IAM policies to access your Amazon EFS file system.

Network considerations with Amazon EFS transfers

VPCs that you use with DataSync must have default tenancy. VPCs with dedicated tenancy aren't supported.

Performance considerations with Amazon EFS transfers

Your Amazon EFS file system's throughput mode can affect transfer duration and file system performance during the transfer. Consider the following:

  • For best results, we recommend using Elastic throughput mode. If you don't use Elastic throughput mode, your transfer might take longer.

  • If you use Bursting throughput mode, the performance of your file system's applications might be affected because DataSync consumes file system burst credits.

  • How you configure DataSync to verify your transferred data can affect file system performance and data access costs.

For more information, see Amazon EFS performance in the Amazon Elastic File System User Guide and the Amazon EFS Pricing page.

Creating your Amazon EFS transfer location

To create the transfer location, you need an existing Amazon EFS file system. If you don't have one, see Getting started with Amazon Elastic File System in the Amazon Elastic File System User Guide.

  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

  3. For Location type, choose Amazon EFS file system.

    You configure this location as a source or destination later.

  4. For File system, choose the Amazon EFS file system that you want to use as a location.

    You configure this location as a source or destination later.

  5. For Mount path, enter a mount path for your Amazon EFS file system.

    This specifies where DataSync reads or writes data (depending on if this is a source or destination location).

    By default, DataSync uses the root directory (or access point if you configure one). You can also specify subdirectories using forward slashes (for example, /path/to/directory).

  6. For Subnet choose a subnet where DataSync creates the network interfaces for managing traffic during your transfer.

    The subnet must be located:

    • In the same VPC as the Amazon EFS file system.

    • In the same Availability Zone as at least one file system mount target.

    Note

    You don't need to specify a subnet that includes a file system mount target.

  7. For Security groups, choose the security groups associated with an Amazon EFS file system's mount target.

    Note

    The security groups that you specify must allow inbound traffic on NFS port 2049. For more information, see Using VPC security groups for Amazon EC2 instances and mount targets in the Amazon Elastic File System User Guide.

  8. For In-transit encryption, choose whether you want DataSync to use Transport Layer Security (TLS) encryption when it copies data to or from your file system.

    Note

    You must enable this setting if you want to configure an access point, IAM role, or both with your location.

  9. (Optional) For EFS access point, choose an access point that DataSync can use to mount your Amazon EFS file system.

  10. (Optional) For IAM role, specify a role that allows DataSync to access your file system.

    For information on creating this role, see Using IAM policies to access your Amazon EFS file system

  11. (Optional) Select Add tag to tag your file system.

    A tag is a key-value pair that helps you manage, filter, and search for your locations.

  12. Choose Create location.

  1. Identify a subnet that has at least one mount target for that file system. You can see all the mount targets and the subnets associated with an EFS file system by using the describe-mount-targets command.

    aws efs describe-mount-targets \ --region aws-region \ --file-system-id file-system-id
    Note

    The AWS Region that you specify is the one where your target S3 bucket or EFS file system is located.

    This command returns information about the target similar to the information shown following.

    { "MountTargets": [ { "OwnerId": "111222333444", "MountTargetId": "fsmt-22334a10", "FileSystemId": "fs-123456ab", "SubnetId": "subnet-f12a0e34", "LifeCycleState": "available", "IpAddress": "11.222.0.123", "NetworkInterfaceId": "eni-1234a044" } ] }
  2. Specify an Amazon EC2 security group that can access the mount target. You can run the following command to find out the security group of the mount target.

    aws efs describe-mount-target-security-groups \ --region aws-region \ --mount-target-id mount-target-id

    The security group that you provide must be able to communicate with the security group on the mount target in the subnet specified.

    The relationship between security group M on the mount target and security group S, which you provide for DataSync to use at this stage, is as follows:

    • Security group M, which you associate with the mount target, must allow inbound access for the TCP protocol on the NFS port (2049) from security group S.

      You can enable an inbound connection either by its IP address (CIDR range) or its security group.

    • Security group S, which you provide to DataSync to access Amazon EFS, should have a rule that enables outbound connections to the NFS port. It enables outbound connections on one of the file system's mount targets.

      You can enable outbound connections either by IP address (CIDR range) or security group.

      For information about security groups and mount targets, see Security groups for Amazon EC2 instances and mount targets in the Amazon Elastic File System User Guide.

  3. Create the location. To create the EFS location, you need the ARNs for your Amazon EC2 subnet, Amazon EC2 security group, and a file system. Because the DataSync API accepts fully qualified ARNs, you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the AWS General Reference.

    Use the following command to create an EFS location.

    aws datasync create-location-efs \ --subdirectory /path/to/your/subdirectory \ --efs-filesystem-arn 'arn:aws:elasticfilesystem:region:account-id:file-system/filesystem-id' \ --ec2-config SecurityGroupArns='arn:aws:ec2:region:account-id:security-group/security-group-id',SubnetArn='arn:aws:ec2:region:account-id:subnet/subnet-id'
Note

The AWS Region that you specify is the one where your target S3 bucket or EFS file system is located.

The command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Using IAM policies to access your Amazon EFS file system

You can configure your Amazon EFS file system with a higher level of security by using IAM policies. In your file system policy, you can specify an IAM role that still allows DataSync to connect with the file system.

Note

To use an IAM role, you must enable TLS for in-transit encryption when creating a DataSync location for your file system.

For more information, see Using IAM to control file system data access in the Amazon Elastic File System User Guide.

Creating an IAM role for DataSync

Create a role with DataSync as the trusted entity.

To create the IAM role
  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the left navigation pane, under Access management, choose Roles, and then choose Create role.

  3. On the Select trusted entity page, for Trusted entity type, choose Custom trust policy.

  4. Paste the following JSON into the policy editor:

    { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" }] }
  5. Choose Next. On the Add permissions page, choose Next.

  6. Give your role a name and choose Create role.

Specify this role when creating the location for your Amazon EFS file system.

Example Amazon EFS file system policy

The following sample IAM policy includes elements that help restrict access to an Amazon EFS file system (identified in the policy as fs-1234567890abcdef0):

  • Principal: Specifies an IAM role that gives DataSync permission to connect to the file system.

  • Action: Gives DataSync root access and allows it to read from and write to the file system.

  • aws:SecureTransport: Requires NFS clients to use TLS when connecting to the file system.

  • elasticfilesystem:AccessPointArn: Allows access to the file system only through a specific access point.

{ "Version": "2012-10-17", "Id": "ExampleEFSFileSystemPolicy", "Statement": [{ "Sid": "AccessEFSFileSystem", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/MyDataSyncRole" }, "Action": [ "elasticfilesystem:ClientMount", "elasticfilesystem:ClientWrite", "elasticfilesystem:ClientRootAccess" ], "Resource": "arn:aws:elasticfilesystem:us-east-1:111122223333:file-system/fs-1234567890abcdef0", "Condition": { "Bool": { "aws:SecureTransport": "true" }, "StringEquals": { "elasticfilesystem:AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:111122223333:access-point/fsap-abcdef01234567890" } } }] }