Configuring how to handle files, objects, and metadata

You can configure how AWS DataSync handles your files, objects, and their associated metadata when transferring between locations.

For example, with recurring transfers, you might want to overwrite files in your destination with changes in the source to keep the locations in sync. You can copy properties such as POSIX permissions for files and folders, tags associated with objects, and access control lists (ACLs).

Transfer mode options

You can configure whether DataSync transfers only the data (including metadata) that's changed following an initial copy or all data every time you run the task. If you're planning on recurring transfers, you might only want to transfer what's changed since your previous task execution.

Option in console	Option in API	Description
Transfer only data that has changed	TransferMode set to CHANGED	After your initial full transfer, DataSync copies only the data and metadata that differs between the source and destination location.
Transfer all data	TransferMode set to ALL	DataSync copies everything in the source to the destination without comparing differences between the locations.

File and object handling options

You can control some aspects of how DataSync treats your files or objects in the destination location. For example, DataSync can delete files in the destination that aren't in the source.

Option in console	Option in API	Description
Keep deleted files	PreserveDeletedFiles	Specifies whether DataSync maintains files or objects in the destination location that don't exist in the source. If you configure your task to delete objects from your Amazon S3 bucket, you might incur minimum storage duration charges for certain storage classes. For detailed information, see Storage class considerations with Amazon S3 transfers. Warning You can't configure your task to delete data in the destination and also transfer all data. When you transfer all data, DataSync doesn't scan your destination location and doesn't know what to delete.
Overwrite files	OverwriteMode	Specifies whether DataSync modifies data in the destination location when the source data or metadata has changed. If you don't configure your task to overwrite data, the destination data isn't overwritten even if the source data differs. If your task overwrites objects, you might incur additional charges for certain storage classes (for example, for retrieval or early deletion). For detailed information, see Storage class considerations with Amazon S3 transfers.

Metadata handling options

DataSync can preserve file and object metadata during a transfer. The metadata that DataSync can preserve depends on the storage systems involved and whether those systems use a similar metadata structure.

Before configuring your task, make sure that you understand how DataSync handles metadata and special files when transferring between your source and destination locations.

Option in console	Option in API	Description
Copy ownership	Gid and Uid	Specifies whether DataSync copies POSIX file and folder ownership, such as the group ID of the file's owners and the user ID of the file's owner.
Copy permissions	PosixPermissions	Specifies whether DataSync copies POSIX permissions for files and folders from the source to the destination.
Copy timestamps	Atime and Mtime	Specifies whether DataSync copies the timestamp metadata from the source to the destination.
Copy object tags	ObjectTags	Specifies whether DataSync preserves the tags associated with your objects when transferring between object storage systems.
Copy ownership, DACLs, and SACLs	SecurityDescriptorCopyFlags set to OWNER_DACL_SACL	DataSync copies the following: The object owner. NTFS discretionary access lists (DACLs), which determine whether to grant access to an object. NTFS system access control lists (SACLs), which are used by administrators to log attempts to access a secured object. Note: SACLs are not copied if you use SMB version 1.0. Copying DACLs and SACLs requires granting specific permissions to the Windows user that DataSync uses to access your location using SMB. For more information, see creating a location for SMB, FSx for Windows File Server, or FSx for ONTAP (depending on the type of location in your transfer).
Copy ownership and DACLs	SecurityDescriptorCopyFlags set to OWNER_DACL	DataSync copies the following: The object owner. DACLs, which determine whether to grant access to an object. DataSync won't copy SACLs when you choose this option.
Do not copy ownership or ACLs	SecurityDescriptorCopyFlags set to NONE	DataSync doesn't copy any ownership or permissions data. The objects that DataSync writes to your destination location are owned by the user whose credentials are provided for DataSync to access the destination. Destination object permissions are determined based on the permissions configured on the destination server.

Configuring file, object, and metadata handling options

You can configure how DataSync handles files, objects, and metadata when creating, editing, or starting your transfer task.

Using the DataSync console

The following instructions describe how to configure file, object, and metadata handling options when creating a task.

1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

2. In the left navigation pane, expand Data transfer, then choose Tasks, and then choose Create task.

3. Configure your task's source and destination locations.

   For more information, see Where can I transfer my data with AWS DataSync?

4. For Transfer mode, choose one of the following options:

   • Transfer only data that has changed

   • Transfer all data

   For more information about these options, see Transfer mode options.

5. Select Keep deleted files if you want DataSync to maintain files or objects in the destination location that don't exist in the source.

   If you don't choose this option and your task deletes objects from your Amazon S3 bucket, you might incur minimum storage duration charges for certain storage classes. For detailed information, see Storage class considerations with Amazon S3 transfers.

   Warning

   You can't deselect this option and enable Transfer all data. When you transfer all data, DataSync doesn't scan your destination location and doesn't know what to delete.

6. Select Overwrite files if you want DataSync to modify data in the destination location when the source data or metadata has changed.

   If your task overwrites objects, you might incur additional charges for certain storage classes (for example, for retrieval or early deletion). For detailed information, see Storage class considerations with Amazon S3 transfers.

   If you don't choose this option, the destination data isn't overwritten even if the source data differs.

7. Under Transfer options, select how you want DataSync to handle metadata. For more information about the options, see Metadata handling options.

   Important

   The options you see in the console depend on your task's source and destination locations. You might have to expand Additional settings to see some of these options.

   • Copy ownership

   • Copy permissions

   • Copy timestamps

   • Copy object tags

   • Copy ownership, DACLs, and SACLs

   • Copy ownership and DACLs

   • Do not copy ownership or ACLs

Using the DataSync API

You can configure file, object, and metadata handling options by using the Options parameter with any of the following operations:

• CreateTask

• StartTaskExecution

• UpdateTask