Monitoring data transfers with Amazon CloudWatch Logs
You can monitor your AWS DataSync transfer by using CloudWatch Logs. We recommend that you configure your task to at least log basic information (such as transfer errors).
Allowing DataSync to upload logs to a CloudWatch log group
To configure logging for your DataSync task, you need a CloudWatch log group that DataSync has permission to send logs to. You set up this access through an AWS Identity and Access Management (IAM) role. How this specifically works depends on your task mode.
Configuring logging for your DataSync task
We recommend that you configure at least some level of logging for your DataSync task.
Before you begin
DataSync needs permission to upload logs to a CloudWatch log group. For more information, see Allowing DataSync to upload logs to a CloudWatch log group.
The following instructions describe how to configure CloudWatch logging when creating a task. You also can configure logging when editing a task.
Open the AWS DataSync console at https://console.aws.amazon.com/datasync/
. In the left navigation pane, expand Data transfer, then choose Tasks, and then choose Create task.
-
Configure your task's source and destination locations.
For more information, see Where can I transfer my data with AWS DataSync?
-
On the Configure settings page, choose a task mode and any other options.
You might be interested in some of the following options:
-
Specify what data to transfer by using a manifest or filters.
-
Configure how to handle file metadata and verify data integrity.
-
-
For Log level, choose one of the following options:
-
Log basic information such as transfer errors – Publish logs with only basic information (such as transfer errors).
-
Log all transferred objects and files – Publish logs for all files or objects that DataSync transfers and performs data-integrity checks on.
-
Don't generate logs
-
-
Do one of the following depending on the task mode you're using to create or specify a CloudWatch log group:
-
Choose Create task.
You're ready to start your task.
-
Copy the following
create-task
command:aws datasync create-task \ --source-location-arn "arn:aws:datasync:
us-east-1
:account-id
:location/location-id
" \ --destination-location-arn "arn:aws:datasync:us-east-1
:account-id
:location/location-id
" \ --task-mode "ENHANCED-or-BASIC
" \ --name "task-name
" \ --options '{"LogLevel": "log-level
"}' \ --cloudwatch-log-group-arn "arn:aws:logs:us-east-1
:account-id
:log-group:log-group-name
:*" -
For
--source-location-arn
, specify the Amazon Resource Name (ARN) of your source location. -
For
--destination-location-arn
, specify the ARN of your destination location.If you're transferring across AWS Regions or accounts, make sure that the ARN includes the other Region or account ID.
-
For
--task-mode
, specifyENHANCED
orBASIC
. -
(Recommended) For
--name
, specify a name for your task that you can remember. -
For
LogLevel
, specify one of the following options:-
BASIC
– Publish logs with only basic information (such as transfer errors). -
TRANSFER
– Publish logs for all files or objects that DataSync transfers and performs data-integrity checks on. -
NONE
– Don't generate logs.
-
-
For -
-cloudwatch-log-group-arn
, specify the ARN of a CloudWatch log group.Important
If your
--task-mode
isENHANCED
, you don't need to specify this option. For more information, see Allowing DataSync to upload logs to a CloudWatch log group. -
Run the
create-task
command.If the command is successful, you get a response that shows you the ARN of the task that you created. For example:
{ "TaskArn": "arn:aws:datasync:us-east-1:111222333444:task/task-08de6e6697796f026" }
You're ready to start your task.
You can configure CloudWatch logging for your task by using the
CloudWatchLogGroupArn
parameter with any of the following
operations:
Viewing DataSync task logs
When you start your task, you can view the task execution's logs by using the CloudWatch console or AWS CLI (among other options). For more information, see the Amazon CloudWatch Logs User Guide.
DataSync provides JSON-structured logs for Enhanced mode tasks. Basic mode tasks have unstructured logs. The following examples show how verification errors display in Enhanced mode logs compared to Basic mode logs.