Understanding data transfer performance counters

When you start a task, AWS DataSync provides counters to help track your data transfer's performance and progress.

Use the following information to understand what each counter represents. You can view these counters in the DataSync console or a DescribeTaskExecution response. Some counters aren't available with every task mode.

Console	DescribeTaskExecution	Task mode support	Description
–	BytesWritten	Enhanced, Basic	The number of logical bytes that DataSync actually writes to the destination location.
Data throughput	–	Enhanced, Basic	The rate at which DataSync writes logical bytes to the destination location. If you're using DescribeTaskExecution, how you calculate this counter depends on your task mode: Enhanced mode: Divide BytesWritten by TotalDuration Basic mode: Divide BytesWritten by TransferDuration
Data transferred	BytesTransferred	Enhanced, Basic	The number of bytes that DataSync sends to the network before compression (if compression is possible). For the number of bytes transferred over the network, see the Network throughput (in console) or BytesCompressed (in DescribeTaskExecution) counter.
Deleted from destination	FilesDeleted	Enhanced, Basic	The number of files, objects, and directories that DataSync actually deletes in your destination location. If you don't configure your task to delete data in the destination that isn't in the source: Deleted from destination doesn't display in the console. FilesDeleted always shows a value of 0.
–	EstimatedBytesToTransfer	Enhanced, Basic	The number of logical bytes that DataSync expects to write to the destination location.
–	EstimatedFilesToDelete	Enhanced, Basic	The number of files, objects, and directories that DataSync expects to delete in your destination location. If you don't configure your task to delete data in the destination that isn't in the source, the value is always 0.
–	EstimatedFilesToTransfer	Enhanced, Basic	The number of files, objects, and directories that DataSync expects to transfer over the network. This value is calculated while DataSync prepares the transfer. How this gets calculated depends primarily on the transfer mode you're using: If transfer mode is set to transfer only data that has changed: The calculation is based on comparing the content of the source and destination locations and determining the difference that needs to be transferred. The difference can include: Anything that's added or modified at the source location. Anything that's in both locations and modified at the destination after an initial transfer (unless you configure your task to not overwrite data in the destination). (Basic mode only) The number of items that DataSync expects to delete (if you configure your task to delete data in the destination). If transfer mode is set to transfer all data: The calculation is based only on the items that DataSync finds at the source location.
File throughput	–	Enhanced, Basic	The rate at which DataSync transfers files, objects, and directories over the network. If you're using DescribeTaskExecution, how you calculate this counter depends on your task mode: Enhanced mode: Divide FilesTransferred by TotalDuration Basic mode: Divide FilesTransferred by TransferDuration
–	FilesFailed	Enhanced	The number of objects that DataSync fails to prepare, transfer, verify, and delete during your task execution. If there are failures, you can see these alongside the Prepared, Transferred, Skipped, and Deleted from destination console counters, respectively.
Listed at source	FilesListed.AtSource	Enhanced	The number of objects that DataSync finds at your source location. With a manifest, DataSync lists only what's in your manifest (and not everything in your source location). With an include filter, DataSync lists only what matches the filter at your source location. With an exclude filter, DataSync lists everything at your source location before applying the filter.
–	FilesListed.AtDestinationForDelete	Enhanced	The number of objects that DataSync finds at your destination location. This counter is only applicable if you configure your task to delete data in the destination that isn't in the source.
Network throughput*	BytesCompressed	Enhanced, Basic	The number of physical bytes that DataSync transfers over the network after compression (if compression is possible). This number is typically less than Data transferred (in console) or BytesTransferred (in DescribeTaskExecution) unless the data isn't compressible. * – For Enhanced mode, Network throughput doesn't display in the console.
Percent compressed	–	Basic	The percentage of transfer data that DataSync compressed before sending it over the network. If you're using DescribeTaskExecution, you can calculate this counter with 1 - BytesCompressed / BytesWritten.
Prepared	FilesPrepared	Enhanced	The number of objects that DataSync will attempt to transfer after comparing your source and destination locations. In the console, this counter can also show you the number of objects that DataSync skips during preparation. For more information, see How DataSync prepares your data transfer. This counter isn't applicable if you configure your task to transfer all data. In that scenario, DataSync copies everything from the source to the destination without comparing differences between locations.
Processing rate	–	Enhanced, Basic	The rate at which DataSync reads files, objects, and directories at your source location. The processing rate is based on several CloudWatch metrics. The exact metrics depend on the task mode you're using. Enhanced mode: FilesListedSource FilesPrepared FilesTransferred FilesVerified Basic mode: FilesPreparedSource FilesPreparedDestination FilesTransferred FilesVerifiedSource FilesVerifiedDestination
Remaining	–	Basic	The remaining number of files, objects, and directories that DataSync expects to transfer over the network. If you're using DescribeTaskExecution, you can calculate this counter by subtracting FilesTransferred from EstimatedFilesToTransfer.
Skipped*	FilesSkipped	Enhanced, Basic	The number of files, objects, and directories that DataSync skips during your transfer. * – For Enhanced mode, Skipped doesn't display in the console. Instead, skipped items are included in the Prepared counter when transferring only the data that has changed or the Transferred counter when transferring all data.
Transferred	FilesTransferred	Enhanced, Basic	The number of files, objects, and directories that DataSync transfers over the network. This value is updated periodically during your task execution when something is read from the source and sent over the network. If DataSync fails to transfer something, this value can be less than EstimatedFilesToTransfer. In some cases, this value can also be greater than EstimatedFilesToTransfer. This counter is implementation-specific for some location types, so don't use it as an exact indication of what's transferring or to monitor your task execution. In the console, this counter can also show you the number of objects that DataSync skips during the transfer of an Enhanced mode task. For more information, see How DataSync transfers your data.
Verified	FilesVerified	Enhanced, Basic	The number of files, objects, and directories that DataSync verifies during your transfer. When you configure your task to verify only transferred data, DataSync doesn't verify directories in some situations or files or objects that fail to transfer.