Configure Task Settings - AWS DataSync

Configure Task Settings

Now that you have an agent and have configured source and destination locations, you configure settings for your task. A task is a set of two locations (source and destination) and a set of options that you use to control the behavior of a task. If you don't specify options when you create a task, AWS DataSync populates the options with service default settings.

To configure settings

  1. On the Configure settings page, enter a name for your task for Task name - optional.

  2. In the Options section, configure the options that you want to use to run your task.

    Available options are as follows:

    • Verify data – Set an optional verification to compare source and destination at the end of the transfer. As DataSync transfers data, it always performs data integrity checks during the transfer. You can enable additional verification to compare source and destination at the end of a transfer. This additional check can verify the entire dataset or only the files that were transferred as part of the task execution. For most use cases, we recommend verifying only the files transferred.

      Data verification options include the following:

      • Verify only the data transferred (recommended) calculates the checksum of transferred files and metadata on the source. It then compares this checksum to the checksum calculated on those files at the destination at the end of the transfer. We recommend this option when transferring to S3 Glacier or S3 Glacier Deep Archive storage classes. For more information, see Considerations When Working with Amazon S3 Storage Classes in DataSync.

      • Verify all data in the destination performs a scan at the end of the transfer of the entire source and entire destination to verify that source and destination are fully synchronized. This option isn't supported when transferring to S3 Glacier or S3 Glacier Deep Archive storage classes. For more information, see Considerations When Working with Amazon S3 Storage Classes in DataSync.

      • Check integrity during the transfer doesn't run additional verification at the end of the transfer. All data transmissions are still integrity-checked with checksum verification during the transfer.

      • Copy ownership and Copy permissions – DataSync preserves metadata between storage systems that have similar metadata structures. The following options apply only when copying data between Network File System (NFS), Amazon EFS, and Amazon S3. When copying data between SMB and Amazon FSx, ownership and permissions are always copied. For more information about metadata preservation using DataSync, see How DataSync Handles Metadata and Special Files.

        • Copy ownership – If you choose this option, DataSync copies file POSIX ownership, such as the group ID of the file's owners and the user ID of the file's owner.

        • Copy permissions – If you choose this option, DataSync copies file POSIX permissions from the source to the destination.

    • Copy timestamps – If you choose this option, DataSync copies the timestamp metadata from the source to the destination.

    • Keep deleted files – If you choose this option, DataSync keeps files in the destination that don't exist in the source file system.

      If your task deletes objects, you might incur minimum storage duration charges for certain storage classes. For detailed information, see Considerations When Working with Amazon S3 Storage Classes in DataSync.

    • Overwrite files – If you choose this option, files at the destination are overwritten by files from the source when the source data or metadata is different. If you don't choose this option, the destination file isn't replaced by the source file, even if the destination file differs from the source file.

      If your task overwrites objects, you might incur additional charges for certain storage classes (for example, for retrieval or early deletion). For detailed information, see Considerations When Working with Amazon S3 Storage Classes in DataSync.

    • Use available or Set bandwidth limit (MiB/s) — If you choose Use available, DataSync uses all the network bandwidth that is available for the transfer. If you choose Set bandwidth limit (MiB/s), you limit the maximum bandwidth that you want DataSync to use for this task.

      You can change bandwidth limits for an in-progress task execution. For more information, see Adjusting Bandwidth Throttling for a Task Execution.

    • Queueing – If you are using a single agent to run multiple tasks, choose this option to make the tasks run in series (first in, first out). For more information, see Queueing Task Executions.

  3. In the Filtering configuration - Optional section, enter a pattern to use as a filter. This pattern defines the criteria for specific files, folders, and objects to exclude from your transfer. To add additional patterns to your filter, choose Add pattern. For detailed information about filtering and syntax for creating patterns, see Filtering the Data Transferred by AWS DataSync.

    If you want to specify an include filter to transfer only specific files, you can do that later when you start a task. For more information, see Including Data in a Transfer.

  4. In the Schedule (optional) section, configure your task to run on a schedule that you specify, with a minimum interval of 1 hour.

  5. For Frequency, configure how frequently you want the task to run. For frequency configuration options, see Configuring a Task Schedule.

  6. In the Tags - optional section, enter Key and Value to tag your task. A tag is a key-value pair that helps you manage, filter, and search for your tasks. We recommend that you create a name tag for your task.

  7. Task logging - optional – If you choose this option, DataSync publishes logs for individual files or objects to the CloudWatch log group that you specify.

    To upload logs to your CloudWatch log group, DataSync requires a resource policy that grants sufficient permissions. If you don't have a policy in the current Region, a check box appears so that you can create the required policy automatically. For an example of such a policy, see Allowing DataSync to Upload Logs to Amazon CloudWatch Log Groups.

    For more information about using log groups and streams, see Working with Log Groups and Log Streams in the Amazon CloudWatch Logs User Guide.

    Use the Log level option to set the level of detail that is logged to CloudWatch Logs. Log level options include the following:

    • Log basic information such as transfer errors – If you choose this option, only basic information such as transfer errors is published to CloudWatch.

    • Log all transferred objects, files, and folders – If you choose this option, log records are published to CloudWatch Logs for all files or objects that the task copies and integrity checks.

    • Do not send logs to CloudWatch – If you choose this option, DataSync logs are not published to CloudWatch.

  8. Choose Next to open the Review page.