Exporting Log Data in Bulk to Amazon S3
You can export log data from your log groups to an Amazon S3 bucket and use this data in custom processing and analysis, or to load onto other systems. You can export your data to an Amazon S3 bucket using the Amazon CloudWatch Logs commands in the AWS CLI, or by using the AWS SDK.
To begin the export process, you must create an Amazon S3 bucket to which the log data will be exported. You can store the exported files in your Amazon S3 bucket and define Amazon S3 lifecycle rules to archive or delete exported files automatically.
You can export logs from multiple log groups or multiple time ranges to the same Amazon S3 bucket. To separate log data for each export task, you can specify an Amazon S3 prefix that will be used as the Amazon S3 key prefix for all exported objects.
Log data can take up to 12 hours to become available for export. For near real-time analysis of log data, see Real-time Processing of Log Data with Subscriptions instead.
Before you begin, become familiar with the following export concepts:
- log group name
The name of the log group associated with an export task. The log data in this log group will be exported to the specified Amazon S3 bucket.
- from (timestamp)
A required timestamp expressed as the number of milliseconds since Jan 1, 1970 00:00:00 UTC. All log events in the log group that were ingested after this time will be exported.
- to (timestamp)
A required timestamp expressed as the number of milliseconds since Jan 1, 1970 00:00:00 UTC. All log events in the log group that were ingested before this time will be exported.
- destination bucket
The name of the Amazon S3 bucket associated with an export task. This bucket is used to export the log data from the specified log group.
- destination prefix
An optional attribute that is used as the S3 key prefix for all exported objects. This helps create a folder-like organization in your bucket.