How DataSync Handles Metadata and Special Files - AWS DataSync
Metadata Copied by DataSyncLinks and Directories Copied by DataSync

How DataSync Handles Metadata and Special Files

DataSync saves metadata and special files when copying to and from file systems. The following section describes how DataSync handles metadata and special files (links and directories) for different storage systems.

Metadata Copied by DataSync

DataSync preserves metadata between storage systems that have similar metadata structures.

When you copy between an NFS server and Amazon EFS – In this case, DataSync copies the following metadata:

  • File and folder modification timestamps

  • User ID and group ID

  • POSIX permissions

When you copy between SMB file share and Amazon FSx for Windows File Server – In this case, DataSync copies the following metadata:

  • File timestamps: access time, modification time, and creation time

  • File owner and file group security identifiers (SIDs)

  • Standard file attributes:

    • Read-only (R)

    • Archive (A)

    • System (S)

    • Hidden (H)

    • Compressed (C)

    • Not content indexed (N)

    • Encrypted (E)

    • Temporary (T)

    • Offline (O)

  • NTFS discretionary access lists (DACLs): access control entries (ACEs) that determine whether to grant access to an object

  • COMING SOON — NTFS system access control lists (SACLs), which are used by administrators to log attempts to access a secured object. Copying SACLs will be added later in 2020. For more information, contact us through AWS Support.

When copying between an NFS server and Amazon S3 – In this case, the following metadata is stored as Amazon S3 user metadata:

  • File and folder modification timestamps

  • User ID and group ID

  • POSIX permissions

The file metadata that is stored in Amazon S3 user metadata is interoperable with file gateways in AWS Storage Gateway. A file gateway enables you to have on-premises file-based access to data stored in Amazon S3 by DataSync. The metadata is also interoperable with Amazon FSx for Lustre.

When DataSync copies objects that contain this metadata back to an NFS server, the file metadata is restored. Restoring metadata requires granting elevated permissions to the NFS server when you are Creating a Location for NFS.

When you copy between storage systems that don’t have similar metadata structure – In this case, DataSync sets metadata using the following rules.

If You Copy This Way This Happens to Metadata

From an SMB share to Amazon EFS or Amazon S3

From Amazon FSx for Windows to an NFS share

Default POSIX metadata is set for all files on the target NFS server or Amazon EFS, or stored in the Amazon S3 object's metadata. This approach includes using the default POSIX user ID and group ID values.

From an NFS share to Amazon FSx for Windows File Server

From Amazon EFS or Amazon S3 to an SMB share

Ownership is set based on the Windows user that was specified in DataSync to access the Amazon FSx or SMB share. Default Windows permissions are assigned based on that file owner.

Default POSIX Metadata Applied by DataSync

When the source and destination don't have a similar metadata structure, or when source metadata is missing, DataSync applies default POSIX metadata.

Specifically, DataSync applies this metadata in these situations:

  • When transferring files from an Amazon S3 location to an Amazon EFS or NFS location, in cases where Amazon S3 objects don't have DataSync POSIX metadata

  • When transferring from an SMB location to an NFS, Amazon S3, or Amazon EFS location

The following table shows the default POSIX metadata and permissions that DataSync applies.

Permission Value

UID

65534

GID

65534

Folder Permission

0755

File Permission

0755

Links and Directories Copied by DataSync

The following section describes how DataSync handles working with copied hard links, symbolic links, and directories in different storage locations.

Hard links

Hard links are preserved when copying between an NFS server and Amazon EFS.

When copying to Amazon S3, each hard link is only transferred once. Separate Amazon S3 objects are created for each copy. If a hard link is unchanged in Amazon S3, it's correctly restored upon transfer to an NFS server or Amazon EFS.

Hard links aren't supported for copying between SMB file shares and Amazon FSx for Windows File Server. If DataSync encounters hard links in such a copy, they are skipped and logged to Amazon CloudWatch Logs. For more information about how DataSync works with CloudWatch Logs, see Allowing DataSync to Upload Logs to Amazon CloudWatch Log Groups.

Symbolic links

Symbolic links are preserved when copying between an NFS server and Amazon EFS.

When copying to Amazon S3, the link target path is stored in the Amazon S3 object. The link is correctly restored upon transfer to an NFS server or Amazon EFS.

Symbolic links aren't supported for copying between SMB file shares and Amazon FSx for Windows File Server. If DataSync encounters symbolic links in such a copy, they are skipped and logged to CloudWatch Logs. For more information about how DataSync works with CloudWatch Logs, see Allowing DataSync to Upload Logs to Amazon CloudWatch Log Groups.

Directories

When copying to or from Amazon S3 buckets, directories are represented as empty objects ending with “/”.