Amazon CloudWatch Logs
User Guide

Export Log Data to Amazon S3 Using the Console

In the following example, you use the Amazon CloudWatch console to export all data from an Amazon CloudWatch Logs log group named my-log-group to an Amazon S3 bucket named my-exported-logs.

Step 1: Create an Amazon S3 Bucket

We recommend that you use a bucket that was created specifically for CloudWatch Logs. However, if you want to use an existing bucket, you can skip to step 2.

Note

The Amazon S3 bucket must reside in the same Region as the log data to export. CloudWatch Logs doesn't support exporting data to Amazon S3 buckets in a different Region.

To create an Amazon S3 bucket

  1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. If necessary, change the Region. From the navigation bar, choose the Region where your CloudWatch Logs reside.

  3. Choose Create Bucket.

  4. For Bucket Name, enter a name for the bucket.

  5. For Region, select the Region where your CloudWatch Logs data resides.

  6. Choose Create.

Step 2: Create an IAM User with Full Access to Amazon S3 and CloudWatch Logs

In the following steps, you create the IAM user with necessary permissions.

To create the necessary IAM user

  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. Choose Users, Add user.

  3. Enter a user name, such as CWLExportUser.

  4. Select both Programmatic access and AWS Management Console access.

  5. Choose either Autogenerated password or Custom password.

  6. Choose Next: Permissions.

  7. Choose Attach existing policies directly, and attach the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user. You can use the search box to find the policies.

  8. Choose Next: Tags, Next: Review, and then Create user.

Step 3: Set Permissions on an Amazon S3 Bucket

By default, all Amazon S3 buckets and objects are private. Only the resource owner, the AWS account that created the bucket, can access the bucket and any objects that it contains. However, the resource owner can choose to grant access permissions to other resources and users by writing an access policy.

When you set the policy, we recommend that you include a randomly generated string as the prefix for the bucket, so that only intended log streams are exported to the bucket.

To set permissions on an Amazon S3 bucket

  1. In the Amazon S3 console, choose the bucket that you created in step 1.

  2. Choose Permissions, Bucket policy.

  3. In the Bucket Policy Editor, add one of the following policies. Change my-exported-logs to the name of your S3 bucket and random-string to a randomly generated string of characters. Be sure to specify the correct Region endpoint for Principal.

    • If the bucket is in your account, add the following policy.

      { "Version": "2012-10-17", "Statement": [ { "Action": "s3:GetBucketAcl", "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } } ] }
    • If the bucket is in a different account, use the following policy instead. It includes an additional statement using the IAM user you created in the previous step.

      { "Version": "2012-10-17", "Statement": [ { "Action": "s3:GetBucketAcl", "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs", "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "Service": "logs.us-west-2.amazonaws.com" } }, { "Action": "s3:PutObject" , "Effect": "Allow", "Resource": "arn:aws:s3:::my-exported-logs/random-string/*", "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } }, "Principal": { "AWS": "arn:aws:iam::SendingAccountID:user/CWLExportUser" } } ] }
  4. Choose Save to set the policy that you just added as the access policy on your bucket. This policy enables CloudWatch Logs to export log data to your Amazon S3 bucket. The bucket owner has full permissions on all of the exported objects.

    Warning

    If the existing bucket already has one or more policies attached to it, add the statements for CloudWatch Logs access to that policy or policies. We recommend that you evaluate the resulting set of permissions to be sure that they're appropriate for the users who will access the bucket.

Step 4: Create an Export Task

In this step, you create the export task for exporting logs from a log group.

To export data to Amazon S3 using the CloudWatch console

  1. Sign in as the IAM user that you created in Step 2: Create an IAM User with Full Access to Amazon S3 and CloudWatch Logs.

  2. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  3. In the navigation pane, choose Logs.

  4. On the Log Groups screen, select the button next to a log group and then choose Actions and then Export data to Amazon S3.

  5. On the Export data to Amazon S3 screen, under Define data to export, set the time range for the data to export using From and To.

  6. If your log group has multiple log streams, you can provide a log stream prefix to limit the log group data to a specific stream. Choose Advanced, and then for Stream prefix, enter the log stream prefix.

  7. Under Choose S3 bucket, choose the account associated with the Amazon S3 bucket.

  8. For S3 bucket name, choose an Amazon S3 bucket.

  9. Choose Advanced, and then for S3 Bucket prefix, enter the randomly generated string that you specified in the bucket policy.

  10. Choose Export data to export your log data to Amazon S3.

  11. To view the status of the log data that you exported to Amazon S3, choose Actions and then View all exports to Amazon S3.