AWS DataSync
User Guide

Monitoring Your Task

You can monitor AWS DataSync using Amazon CloudWatch, which collects and processes raw data from DataSync into readable, near real-time metrics. These statistics are retained for a period of 15 months, so that you can access historical information and gain a better perspective on how DataSync. By default, DataSync metric data is automatically sent to CloudWatch at 5-minute periods. For more information about CloudWatch, see What Are Amazon CloudWatch, CloudWatch Events, and CloudWatch Logs? in the Amazon CloudWatch User Guide.

Accessing Amazon CloudWatch Metrics for DataSync

Amazon CloudWatch provides metrics that you can use to get information about DataSync performance. You can see CloudWatch metrics for DataSync in many ways. You can view them through the CloudWatch console, or you can access them using the CloudWatch CLI or the CloudWatch API. You can also see these metrics on the task execution details page in the AWS DataSync console. For information about how to use CloudWatch metrics, see Using Amazon CloudWatch Metrics in the Amazon CloudWatch User Guide.

DataSync CloudWatch Metrics

The AWS/DataSync namespace includes the following metrics.

These statistics are retained for a period of 15 months.

Metric Description

BytesVerifiedSource

The total number of bytes of data that are verified at the source location.

Units: Bytes

BytesPreparedSource

The total number of bytes of data that are prepared at the source location.

Unit: Bytes

FilesVerifiedSource

The total number of files that are verified at the source location.

Unit: Count

FilesPreparedSource

The total number of files that are prepared at the source location.

Unit: Count

BytesVerifiedDestination

The total number of bytes of data that are verified at the destination location.

Unit: Bytes

BytesPreparedDestination

The total number of bytes of data that are prepared at the destination location.

Unit: Bytes

FilesVerifiedDestination

The total number of files that are verified at the destination location.

Unit: Count

FilesPreparedDestination

The total number of files that are prepared at the destination location.

Unit: Count

FilesTransferred

The actual number of files or metadata that were transferred over the network. This value is calculated and updated on an ongoing basis during the TRANSFERRING phase. It's updated periodically when each file is read from the source location and sent over the network.

If failures occur during a transfer, this value can be less than EstimatedFilesToTransfer. This value can also be greater than EstimatedFilesTransferred in some cases. This element is implementation-specific for some location types, so don't use it as an indicator for a correct file number or to monitor your task execution.

Unit: Count

BytesTransferred

The total number of bytes that are transferred over the network when the agent reads from the source location to the destination location.

Unit: Bytes

BytesWritten

The total logical size of all files that have been transferred to the destination location.

Unit: Bytes

CloudWatch Events for DataSync

CloudWatch events describe changes in DataSync resources. You can set up rules to match these events and route them to one or more target functions or streams.

The following CloudWatch events are available for AWS DataSync.

Event Description
State Changes for an Agent For details, see Understanding Agent Statuses.
ONLINE The agent is configured properly and is available to use.
OFFLINE The agent's VM is turned off or the agent is in an unhealthy state.
State Changes for a Location
ADDING DataSync is adding a location.
AVAILABLE The location is crated and is available to use.
State Changes for a Task For details, see Understanding Task Creation Statuses.
CREATING DataSync attempts to mount the Network File System (NFS) location and create the task.
RUNNING DataSync has mounted the source and it is functioning properly.
AVAILABLE The task is configured properly and is available to be started.
UNAVAILABLE The task is not configured properly and is not available for use. If an agent that is associated with a source (NFS) location goes offline, the task transitions to the UNAVAILABLE status.
State Changes for a Task Execution For details, see Understanding Task Execution Statuses.
LAUNCHING DataSync is initializing the task execution.
PREPARING DataSync is computing which files need to be transferred.
TRANSFERRING DataSync is performing the actual transfer of your data to AWS
VERIFYING DataSync performs a full data and metadata integrity verification to ensure that the data in your destination is an exact copy of your source.
SUCCESS The transfer is successful.
ERROR The sync has failed.

DataSync Dimensions

DataSync metrics use the AWS/DataSync namespace and provide metrics for the following dimensions:

  • AgentId—the unique ID of the agent.

  • TaskId—the unique ID of the task. It takes the form task-01234567890abcdef.

Allowing DataSync to Upload Logs to Amazon CloudWatch Log Groups

You can use CloudWatch Log Groups to monitor and debug your tasks. For DataSync to upload logs to your CloudWatch Log Group, DataSync requires a resource policy that grants sufficient permissions.

The following is an example of the simplest policy that grants such permissions.

{ "Statement": [ {recorded for a period of 15 months and the data is "Sid": "DataSyncLogsToCloudWatchLogs", "Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:CreateLogStream" ], "Principal": { "Service": "datasync.amazonaws.com" }, "Resource": "*" } ], "Version": "2012-10-17" }

The following code snippet shows an example of how to use the AWS CLI to attach the policy to an Amazon CloudWatch Log Group.

$ POLICY_FILE=$(mktemp -t datasync.policy.XXXXXX.json) $ cat <<EOF > ${POLICY_FILE} { "Statement": [ { "Sid": "DataSyncLogsToCloudWatchLogs", "Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:CreateLogStream" ], "Principal": { "Service": "datasync.amazonaws.com" }, "Resource": "*" } ], "Version": "2012-10-17" } EOF $ aws logs put-resource-policy --policy-name trustDataSync --policy-document file://${POLICY_FILE}

For information about CloudWatch Log Groups, see Working with Log Groups and Log Streams in the Amazon CloudWatch User Guide.

.