Export log data to Amazon S3 using the console - Amazon CloudWatch Logs

Export log data to Amazon S3 using the console

In the following example, you use the Amazon CloudWatch console to export all data from an Amazon CloudWatch Logs log group named my-log-group to an Amazon S3 bucket named my-exported-logs.

Exporting log data to S3 buckets that are encrypted by AWS KMS is supported.

Step 1: Create an Amazon S3 bucket

We recommend that you use a bucket that was created specifically for CloudWatch Logs. However, if you want to use an existing bucket, you can skip to step 2.

Note

The S3 bucket must reside in the same Region as the log data to export. CloudWatch Logs doesn't support exporting data to S3 buckets in a different Region.

To create an S3 bucket

  1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. If necessary, change the Region. From the navigation bar, choose the Region where your CloudWatch Logs reside.

  3. Choose Create Bucket.

  4. For Bucket Name, enter a name for the bucket.

  5. For Region, select the Region where your CloudWatch Logs data resides.

  6. Choose Create.

Step 2: Create an IAM user with full access to Amazon S3 and CloudWatch Logs

In the following steps, you create the IAM user with necessary permissions.

To create the necessary IAM user

  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. Choose Users, Add user.

  3. Enter a user name, such as CWLExportUser.

  4. Select both Programmatic access and AWS Management Console access.

  5. Choose either Autogenerated password or Custom password.

  6. Choose Next: Permissions.

  7. Choose Attach existing policies directly, and attach the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user. You can use the search box to find the policies.

  8. Choose Next: Tags, Next: Review, and then Create user.

Step 3: Set permissions on an S3 bucket

By default, all S3 buckets and objects are private. Only the resource owner, the AWS account that created the bucket, can access the bucket and any objects that it contains. However, the resource owner can choose to grant access permissions to other resources and users by writing an access policy.

When you set the policy, we recommend that you include a randomly generated string as the prefix for the bucket, so that only intended log streams are exported to the bucket.

Important

To make exports to S3 buckets more secure, we now require you to specify the list of source accounts that are allowed to export log data to your S3 bucket. In the example below, the list of these account IDs is specified in the aws:SourceAccount key.

We recommend that you also include the account ID of the account where the S3 bucket is created, to allow export within the same account.

To set permissions on an Amazon S3 bucket

  1. In the Amazon S3 console, choose the bucket that you created in step 1.

  2. Choose Permissions, Bucket policy.

  3. In the Bucket Policy Editor, add the following policy. Change my-exported-logs to the name of your S3 bucket and random-string to a randomly generated string of characters. Be sure to specify the correct Region endpoint for Principal.

    • { "Version": "2012-10-17", "Statement": [ { "Action": "s3:GetBucketAcl", "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control", "aws:SourceAccount": [ "AccountId1", "AccountId2", ... ] } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } } ] }
  4. Choose Save to set the policy that you just added as the access policy on your bucket. This policy enables CloudWatch Logs to export log data to your S3 bucket. The bucket owner has full permissions on all of the exported objects.

    Warning

    If the existing bucket already has one or more policies attached to it, add the statements for CloudWatch Logs access to that policy or policies. We recommend that you evaluate the resulting set of permissions to be sure that they're appropriate for the users who will access the bucket.

(Optional) Step 4: Exporting to a bucket encrypted with SSE-KMS

This step is necessary only if you are exporting to an S3 bucket that uses server-side encryption with AWS KMS keys. This encryption is known as SSE-KMS.

To export to a bucket encrypted with SSE-KMS

  1. Open the AWS KMS console at https://console.aws.amazon.com/kms.

  2. To change the AWS Region, use the Region selector in the upper-right corner of the page.

  3. In the left navigation bar, choose Customer managed keys.

    Choose Create Key.

  4. For Key type, choose Symmetric.

  5. For Key usage, choose Encrypt and decrypt and then choose Next.

  6. Under Add labels, enter an alias for the key and optionally add a description or tags. Then choose Next.

  7. Under Key administrators, select who can administer this key, and then choose Next.

  8. Under Define key usage permissions, make no changes and choose Next.

  9. Review the settings and choose Finish.

  10. Back at the Customer managed keys page, choose the name of the key that you just created.

  11. Choose the Key policy tab and choose Switch to policy view.

  12. In the Key policy section, choose Edit.

  13. Add the following statement to the key policy statement list. When you do, replace Region with the Region of your logs and replace account-ARN with the ARN of the account that owns the KMS key.

    { "Version": "2012-10-17", "Statement": [ { "Sid": "Allow CWL Service Principal usage", "Effect": "Allow", "Principal": { "Service": "logs.Region.amazonaws.com" }, "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "*" }, { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": "account-ARN" }, "Action": [ "kms:GetKeyPolicy*", "kms:PutKeyPolicy*", "kms:DescribeKey*", "kms:CreateAlias*", "kms:ScheduleKeyDeletion*", "kms:Decrypt" ], "Resource": "*" } ] }
  14. Choose Save changes.

  15. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  16. Find the bucket that you created in Step 1: Create an S3 bucket and choose the bucket name.

  17. Choose the Properties tab. Then, under Default Encryption, choose Edit.

  18. Under Server-side Encryption, choose Enable.

  19. Under Encryption type, choose AWS Key Management Service key (SSE-KMS).

  20. Choose Choose from your AWS KMS keys and find the key that you created.

  21. For Bucket key, choose Enable.

  22. Choose Save changes.

Step 5: Create an export task

In this step, you create the export task for exporting logs from a log group.

To export data to Amazon S3 using the CloudWatch console

  1. Sign in as the IAM user that you created in Step 2: Create an IAM user with full access to Amazon S3 and CloudWatch Logs.

  2. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  3. In the navigation pane, choose Log groups.

  4. On the Log Groups screen, choose the name of the log group.

  5. Choose Actions, Export data to Amazon S3.

  6. On the Export data to Amazon S3 screen, under Define data export, set the time range for the data to export using From and To.

  7. If your log group has multiple log streams, you can provide a log stream prefix to limit the log group data to a specific stream. Choose Advanced, and then for Stream prefix, enter the log stream prefix.

  8. Under Choose S3 bucket, choose the account associated with the S3 bucket.

  9. For S3 bucket name, choose an S3 bucket.

  10. For S3 Bucket prefix, enter the randomly generated string that you specified in the bucket policy.

  11. Choose Export to export your log data to Amazon S3.

  12. To view the status of the log data that you exported to Amazon S3, choose Actions and then View all exports to Amazon S3.