How DataSync handles metadata and special files

DataSync saves metadata and special files when copying to and from file systems. The following section describes how DataSync handles metadata and special files (links and directories) for different storage systems.

Note

DataSync only copies your storage contents, and does not copy the configuration of the storage itself. For example, when copying objects between Amazon S3 buckets, DataSync does not copy any bucket-level settings or permissions. Likewise, when copying between SMB or NFS shares, DataSync does not copy any file-system level settings or permissions.

Metadata copied by DataSync

DataSync preserves metadata between storage systems that have similar metadata structures.

When copying between self-managed NFS or Amazon EFS, and Amazon EFS – In this case, DataSync can copy the following metadata:

• File and folder modification timestamps

• User ID and group ID

• POSIX permissions

When copying between self-managed SMB or Amazon FSx for Windows File Server, and Amazon FSx for Windows File Server – In this case, DataSync can copy the following metadata:

• File timestamps: access time, modification time, and creation time

• File owner security identifier (SID)

• Standard file attributes:

  • Read-only (R)

  • Archive (A)

  • System (S)

  • Hidden (H)

  • Compressed (C)

  • Not content indexed (N)

  • Encrypted (E)

  • Temporary (T)

  • Offline (O)

  Note

  DataSync tries to copy the Archive and Compressed attributes. However, these attributes might not be applied on the destination, so they are ignored during the verification phase.

• NTFS discretionary access lists (DACLs) which determine whether to grant access to an object.

• NTFS system access control lists (SACLs) which are used by administrators to log attempts to access a secured object.

When copying between self-managed NFS or Amazon EFS, and Amazon S3 – In this case, the following metadata is stored as Amazon S3 user metadata:

• File and folder modification timestamps

• User ID and group ID

• POSIX permissions

The file metadata that is stored in Amazon S3 user metadata is interoperable with NFS shares on file gateways in AWS Storage Gateway. A file gateway enables low latency access from on-premises to data that was copied to Amazon S3 by DataSync. The metadata is also interoperable with Amazon FSx for Lustre.

When DataSync copies objects that contain this metadata back to an NFS server, the file metadata is restored. Restoring metadata requires granting elevated permissions to the NFS server when you are Creating a location for NFS.

When copying between self-managed object storage and Amazon S3, or between two Amazon S3 buckets – In this case, DataSync copies object metadata and tags.

Note

DataSync does not copy other object information, such as object access control lists (ACLs) or prior object versions.

When you copy between storage systems that don't have similar metadata structure – In this case, DataSync sets metadata using the following rules.

If you copy this way	This happens to metadata
From an SMB share to Amazon EFS or Amazon S3 From Amazon FSx for Windows File Server to an NFS share	Default POSIX metadata is set for all files on the target NFS server or Amazon EFS, or stored in the Amazon S3 object's metadata. This approach includes using the default POSIX user ID and group ID values.
From an NFS share to Amazon FSx for Windows File Server From Amazon EFS or Amazon S3 to an SMB share	Ownership is set based on the Windows user that was specified in DataSync to access the Amazon FSx or SMB share. Default Windows permissions are assigned based on that file owner.

Default POSIX metadata applied by DataSync

When the source and destination don't have a similar metadata structure, or when source metadata is missing, DataSync applies default POSIX metadata.

Specifically, DataSync applies this metadata in these situations:

• When transferring files from an Amazon S3 or self-managed object storage location to an Amazon EFS or NFS location, in cases where Amazon S3 objects don't have DataSync POSIX metadata

• When transferring from an SMB location to an NFS, Amazon S3, or Amazon EFS location

The following table shows the default POSIX metadata and permissions that DataSync applies.

Permission	Value
UID	65534
GID	65534
Folder Permission	0755
File Permission	0644

Links and directories copied by DataSync

The following section describes how DataSync handles working with copied hard links, symbolic links, and directories in different storage locations.

Hard links

Hard links are preserved when copying between an NFS server and Amazon EFS.

When copying to Amazon S3, each hard link is only transferred once. Separate Amazon S3 objects are created for each copy. If a hard link is unchanged in Amazon S3, it's correctly restored upon transfer to an NFS server or Amazon EFS.

Hard links aren't supported for copying between SMB file shares and Amazon FSx for Windows File Server. If DataSync encounters hard links in such a copy, they are skipped and logged to Amazon CloudWatch Logs. For more information about how DataSync works with CloudWatch Logs, see Allowing DataSync to upload logs to Amazon CloudWatch log groups.

Symbolic links

Symbolic links are preserved when copying between an NFS server and Amazon EFS.

When copying to Amazon S3, the link target path is stored in the Amazon S3 object. The link is correctly restored upon transfer to an NFS server or Amazon EFS.

Symbolic links aren't supported for copying between SMB file shares and Amazon FSx for Windows File Server. If DataSync encounters symbolic links in such a copy, they are skipped and logged to CloudWatch Logs. For more information about how DataSync works with CloudWatch Logs, see Allowing DataSync to upload logs to Amazon CloudWatch log groups.

Directories

When copying to or from Amazon S3 buckets, directories are represented as empty objects ending with "/".