Exporting log data to Amazon S3 - Amazon CloudWatch Logs

Exporting log data to Amazon S3

Export log data from your log groups to an Amazon S3 bucket and use this data in custom processing and analysis, or to load onto other systems.

You can do the following:

  • Export log data to S3 buckets that are encrypted by AWS Key Management Service (AWS KMS)

  • Export log data to S3 buckets that have S3 Object Lock enabled with a retention period

To begin the export process, you must create an S3 bucket to store the exported log data. You can store the exported files in your S3 bucket and define Amazon S3 lifecycle rules to archive or delete exported files automatically.

You can export to S3 buckets that are encrypted with AES-256. However, exporting to S3 buckets that are encrypted with SSE-KMS is not supported. For more information, see Enabling Amazon S3 default bucket encryption.

You can export logs from multiple log groups or multiple time ranges to the same S3 bucket. To separate log data for each export task, you can specify a prefix that will be used as the Amazon S3 key prefix for all exported objects.

Note

Time-based sorting on chunks of log data inside an exported file is not guaranteed. You can sort the exported log field data by using Linux utilities.

Log data can take up to 12 hours to become available for export. For near real-time analysis of log data, see Analyzing log data with CloudWatch Logs Insights or Real-time processing of log data with subscriptions instead.

Note

Starting on February 15, 2019, the export to Amazon S3 feature requires callers to have s3:PutObject access to the destination bucket.

Concepts

Before you begin, become familiar with the following export concepts:

log group name

The name of the log group associated with an export task. The log data in this log group will be exported to the specified S3 bucket.

from (timestamp)

A required timestamp expressed as the number of milliseconds since Jan 1, 1970 00:00:00 UTC. All log events in the log group that were ingested after this time will be exported.

to (timestamp)

A required timestamp expressed as the number of milliseconds since Jan 1, 1970 00:00:00 UTC. All log events in the log group that were ingested before this time will be exported.

destination bucket

The name of the S3 bucket associated with an export task. This bucket is used to export the log data from the specified log group.

destination prefix

An optional attribute that is used as the Amazon S3 key prefix for all exported objects. This helps create a folder-like organization in your bucket.