Copy trail events to an event data store - AWS CloudTrail

Copy trail events to an event data store

You can copy trail events to a CloudTrail Lake event data store to create a point-in-time snapshot of events logged to the trail. Copying a trail's events does not interfere with the trail's ability to log events and does not modify the trail in any way.

You can copy trail events to an existing event data store configured for CloudTrail events, or you can create a new CloudTrail event data store and choose the Copy trail events option as part of event data store creation. For more information about copying trail events to an existing event data store, see Copy trail events to an existing event data store. For more information about creating a new event data store, see Create an event data store for CloudTrail events with the console.

If you are copying trail events to an organization event data store, you must use the management account for the organization. You cannot copy trail events using the delegated administrator account for an organization.

CloudTrail Lake event data stores incur charges. When you create an event data store, you choose the pricing option you want to use for the event data store. The pricing option determines the cost for ingesting and storing events, and the default and maximum retention period for the event data store. For information about CloudTrail pricing and managing Lake costs, see AWS CloudTrail Pricing and Managing CloudTrail Lake costs.

When you copy trail events to a CloudTrail Lake event data store, you incur charges based on the amount of uncompressed data the event data store ingests.

When you copy trail events to CloudTrail Lake, CloudTrail unzips the logs that are stored in gzip (compressed) format and then copies the events contained in the logs to your event data store. The size of the uncompressed data could be greater than the actual S3 storage size. To get a general estimate of the size of the uncompressed data, you can multiply the size of the logs in the S3 bucket by 10.

You can reduce costs by specifying a narrower time range for the copied events. If you are planning to only use the event data store to query your copied events, you can turn off event ingestion to avoid incurring charges on future events. For more information, see AWS CloudTrail Pricing and Managing CloudTrail Lake costs.

Scenarios

The following table describes some common scenarios for copying trail events and how you accomplish each scenario using the console.

Scenario How do I accomplish this in the console?

Analyze and query historical trail events in CloudTrail Lake without ingesting new events

Create a new event data store and choose the Copy trail events option as part of event data store creation. When creating the event data store, deselect Ingest events (step 15 of the procedure) to ensure the event data store contains only the historical events for your trail and no future events.

Replace your existing trail with a CloudTrail Lake event data store

Create an event data store with the same event selectors as your trail to ensure that the event data store has the same coverage as your trail.

To avoid duplicating events between the source trail and destination event data store, choose a date range for the copied events that is earlier than the creation of the event data store.

After your event data store is created, you can turn off logging for the trail to avoid additional charges.

Considerations for copying trail events

Consider the following factors when copying trail events.

  • When copying trail events, CloudTrail uses the S3 GetObject API operation to retrieve the trail events in the source S3 bucket. There are some S3 archived storage classes, such as S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive, S3 Outposts, and S3 Intelligent-Tiering Deep Archive tiers that are not accessible by using GetObject. To copy trail events stored in these archived storage classes, you must first restore a copy using the S3 RestoreObject operation. For information about restoring archived objects, see Restoring Archived Objects in the Amazon S3 User Guide.

  • When you copy trail events to an event data store, CloudTrail copies all trail events regardless of the configuration of the destination event data store's event types, advanced event selectors, or AWS Region.

  • Before copying trail events to an existing event data store, be sure the event data store's pricing option and retention period are configured appropriately for your use case.

    • Pricing option: The pricing option determines the cost for ingesting and storing events. For more information about pricing options, see AWS CloudTrail Pricing and Event data store pricing options.

    • Retention period: The retention period determines how long event data is kept in the event data store. CloudTrail only copies trail events that have an eventTime within the event data store’s retention period. To determine the appropriate retention period, take the sum of the oldest event you want to copy in days and the number of days you want to retain the events in the event data store (retention period = oldest-event-in-days + number-days-to-retain). For example, if the oldest event you're copying is 45 days old and you want to keep the events in the event data store for a further 45 days, you would set the retention period to 90 days.

  • If you are copying trail events to an event data store for investigation and do not want to ingest any future events, you can stop ingestion on the event data store. When creating the event data store, deselect the Ingest events option (step 15 of the procedure) to ensure the event data store contains only the historical events for your trail and no future events.

  • Before copying trail events, disable any access control lists (ACLs) attached to the source S3 bucket, and update the S3 bucket policy for the destination event data store. For more information about updating the S3 bucket policy, see Amazon S3 bucket policy for copying trail events. For more information about disabling ACLs, see Controlling ownership of objects and disabling ACLs for your bucket in the Amazon S3 User Guide.

  • CloudTrail only copies trail events from Gzip compressed log files that are in the source S3 bucket. CloudTrail does not copy trail events from uncompressed log files or log files that were compressed using a format other than Gzip.

  • To avoid duplicating events between the source trail and destination event data store, choose a time range for the copied events that is earlier than the creation of the event data store.

  • By default, CloudTrail only copies CloudTrail events contained in the S3 bucket's CloudTrail prefix and the prefixes inside the CloudTrail prefix, and does not check prefixes for other AWS services. If you want to copy CloudTrail events contained in another prefix, you must choose the prefix when you copy trail events.

  • To copy trail events to an organization event data store, you must use the management account for the organization. The delegated administrator account cannot copy trail events to an organization event data store.

Required permissions for copying trail events

Before copying trail events, ensure you have all the required permissions for your IAM role. You only need to update the IAM role permissions if you choose an existing IAM role to copy trail events. If you choose to create a new IAM role, CloudTrail provides all necessary permissions for the role.

If the source S3 bucket uses a KMS key for data encryption, ensure that the KMS key policy allows CloudTrail to decrypt data in the bucket. If the source S3 bucket uses multiple KMS keys, you must update each key's policy to allow CloudTrail to decrypt data in the bucket.

IAM permissions for copying trail events

When copying trail events, you have the option to create a new IAM role, or use an existing IAM role. When you choose a new IAM role, CloudTrail creates an IAM role with the required permissions and no further action is required on your part.

If you choose an existing role, ensure the IAM role's policies allow CloudTrail to copy trail events from the source S3 bucket. This section provides examples of the required IAM role permission and trust policies.

The following example provides the permissions policy, which allows CloudTrail to copy trail events from the source S3 bucket. Replace myBucketName, myAccountID, region, prefix, and eventDataStoreId with the appropriate values for your configuration. The myAccountID is the AWS account ID used for CloudTrail Lake, which may not be the same as the AWS account ID for the S3 bucket.

Replace key-region, keyAccountID, and keyID with the values for the KMS key used to encrypt the source S3 bucket. You can omit the AWSCloudTrailImportKeyAccess statement if the source S3 bucket does not use a KMS key for encryption.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AWSCloudTrailImportBucketAccess", "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketAcl"], "Resource": [ "arn:aws:s3:::myBucketName" ], "Condition": { "StringEquals": { "aws:SourceAccount": "myAccountID", "aws:SourceArn": "arn:aws:cloudtrail:region:myAccountID:eventdataStore/eventDataStoreId" } } }, { "Sid": "AWSCloudTrailImportObjectAccess", "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": [ "arn:aws:s3:::myBucketName/prefix", "arn:aws:s3:::myBucketName/prefix/*" ], "Condition": { "StringEquals": { "aws:SourceAccount": "myAccountID", "aws:SourceArn": "arn:aws:cloudtrail:region:myAccountID:eventdataStore/eventDataStoreId" } } }, { "Sid": "AWSCloudTrailImportKeyAccess", "Effect": "Allow", "Action": ["kms:GenerateDataKey","kms:Decrypt"], "Resource": [ "arn:aws:kms:key-region:keyAccountID:key/keyID" ] } ] }

The following example provides the IAM trust policy, which allows CloudTrail to assume an IAM role to copy trail events from the source S3 bucket. Replace myAccountID, region, and eventDataStoreArn with the appropriate values for your configuration. The myAccountID is the AWS account ID used for CloudTrail Lake, which may not be the same as the AWS account ID for the S3 bucket.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "aws:SourceAccount": "myAccountID", "aws:SourceArn": "arn:aws:cloudtrail:region:myAccountID:eventdataStore/eventDataStoreId" } } } ] }

Amazon S3 bucket policy for copying trail events

By default, Amazon S3 buckets and objects are private. Only the resource owner (the AWS account that created the bucket) can access the bucket and objects it contains. The resource owner can grant access permissions to other resources and users by writing an access policy.

Before you copy trail events, you must update the S3 bucket policy to allow CloudTrail to copy trail events from the source S3 bucket.

You can add the following statement to the S3 bucket policy to grant these permissions. Replace roleArn and myBucketName with the appropriate values for your configuration.

{ "Sid": "AWSCloudTrailImportBucketAccess", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketAcl", "s3:GetObject" ], "Principal": { "AWS": "roleArn" }, "Resource": [ "arn:aws:s3:::myBucketName", "arn:aws:s3:::myBucketName/*" ] },

KMS key policy for decrypting data in the source S3 bucket

If the source S3 bucket uses a KMS key for data encryption, ensure the KMS key policy provides CloudTrail with the kms:Decrypt and kms:GenerateDataKey permissions required to copy trail events from an S3 bucket with SSE-KMS encryption enabled. If your source S3 bucket uses multiple KMS keys, you must update each key's policy. Updating the KMS key policy allows CloudTrail to decrypt data in the source S3 bucket, run validation checks to ensure that events conform to CloudTrail standards, and copy events into the CloudTrail Lake event data store.

The following example provides the KMS key policy, which allows CloudTrail to decrypt the data in the source S3 bucket. Replace roleArn, myBucketName, myAccountID, region, and eventDataStoreId with the appropriate values for your configuration. The myAccountID is the AWS account ID used for CloudTrail Lake, which may not be the same as the AWS account ID for the S3 bucket.

{ "Sid": "AWSCloudTrailImportDecrypt", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Principal": { "AWS": "roleArn" }, "Resource": "*", "Condition": { "StringLike": { "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::myBucketName/*" }, "StringEquals": { "aws:SourceAccount": "myAccountID", "aws:SourceArn": "arn:aws:cloudtrail:region:myAccountID:eventdataStore/eventDataStoreId" } } }

Copy trail events to an existing event data store

Use the following procedure to copy trail events to an existing event data store. For information about how to create a new event data store, see Create an event data store for CloudTrail events with the console.

Note

Before copying trail events to an existing event data store, be sure the event data store's pricing option and retention period are configured appropriately for your use case.

  • Pricing option: The pricing option determines the cost for ingesting and storing events. For more information about pricing options, see AWS CloudTrail Pricing and Event data store pricing options.

  • Retention period: The retention period determines how long event data is kept in the event data store. CloudTrail only copies trail events that have an eventTime within the event data store’s retention period. To determine the appropriate retention period, take the sum of the oldest event you want to copy in days and the number of days you want to retain the events in the event data store (retention period = oldest-event-in-days + number-days-to-retain). For example, if the oldest event you're copying is 45 days old and you want to keep the events in the event data store for a further 45 days, you would set the retention period to 90 days.

To copy trail events to an event data store
  1. Sign in to the AWS Management Console and open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.

  2. From the navigation pane, under Lake, choose Event data stores.

  3. Choose Copy trail events.

  4. On the Copy trail events page, for Event source, choose the trail that you want to copy. By default, CloudTrail only copies CloudTrail events contained in the S3 bucket's CloudTrail prefix and the prefixes inside the CloudTrail prefix, and does not check prefixes for other AWS services. If you want to copy CloudTrail events contained in another prefix, choose Enter S3 URI, and then choose Browse S3 to browse to the prefix. If the source S3 bucket for the trail uses a KMS key for data encryption, ensure that the KMS key policy allows CloudTrail to decrypt the data. If your source S3 bucket uses multiple KMS keys, you must update each key's policy to allow CloudTrail to decrypt the data in the bucket. For more information about updating the KMS key policy, see KMS key policy for decrypting data in the source S3 bucket.

    The S3 bucket policy must grant CloudTrail access to copy trail events from your S3 bucket. For more information about updating the S3 bucket policy, see Amazon S3 bucket policy for copying trail events.

  5. For Specify a time range of events, choose the time range for copying the events. CloudTrail checks the prefix and log file name to verify the name contains a date between the chosen start and end date before attempting to copy trail events. You can choose a Relative range or an Absolute range. To avoid duplicating events between the source trail and destination event data store, choose a time range that is earlier than the creation of the event data store.

    Note

    CloudTrail only copies trail events that have an eventTime within the event data store’s retention period. For example, if an event data store’s retention period is 90 days, then CloudTrail will not copy any trail events with an eventTime older than 90 days.

    • If you choose Relative range, you can choose to copy events logged in the last 6 months, 1 year, 2 years, 7 years, or a custom range. CloudTrail copies the events logged within the chosen time period.

    • If you choose Absolute range, you can choose a specific start and end date. CloudTrail copies the events that occurred between the chosen start and end dates.

  6. For Delivery location, choose the destination event data store from the drop-down list.

  7. For Permissions, choose from the following IAM role options. If you choose an existing IAM role, verify that the IAM role policy provides the necessary permissions. For more information about updating the IAM role permissions, see IAM permissions for copying trail events.

    • Choose Create a new role (recommended) to create a new IAM role. For Enter IAM role name, enter a name for the role. CloudTrail automatically creates the necessary permissions for this new role.

    • Choose Use a custom IAM role ARN to use a custom IAM role that is not listed. For Enter IAM role ARN, enter the IAM ARN.

    • Choose an existing IAM role from the drop-down list.

  8. Choose Copy events.

  9. You are prompted to confirm. When you are ready to confirm, choose Copy trail events to Lake, and then choose Copy events.

  10. On the Copy details page, you can see the copy status and review any failures. When a trail event copy completes, its Copy status is set to either Completed if there were no errors, or Failed if errors occurred.

    Note

    Details shown on the event copy details page are not in real-time. The actual values for details such as Prefixes copied may be higher than what is shown on the page. CloudTrail updates the details incrementally over the course of the event copy.

  11. If the Copy status is Failed, fix any errors shown in Copy failures, and then choose Retry copy. When you retry a copy, CloudTrail resumes the copy at the location where the failure occurred.

For more information about viewing the details of a trail event copy, see Event copy details.