Managing CloudTrail Lake costs - AWS CloudTrail

Managing CloudTrail Lake costs

AWS CloudTrail Lake event data stores and queries incur charges. As a best practice, we recommend using AWS services and tools that can help you manage CloudTrail costs. You can also configure event data stores in ways that capture the data you need while remaining cost-effective. For information about CloudTrail pricing, see AWS CloudTrail Pricing.

Event data store pricing options

When you create an event data store, you choose the pricing option that you want to use for the event data store. The pricing option determines the cost for ingesting and storing events, and the default and maximum retention periods for the event data store.

The following table describes the available pricing options. The table shows the Pricing option in the console and the corresponding BillingMode value for the API, and lists the default and maximum retention period for each option.

Pricing option (console) BillingMode (API) Description

One-year extendable retention pricing

EXTENDABLE_RETENTION_PRICING

Recommended if you expect to ingest less than 25 TB of event data per month and want a flexible retention period of up to 10 years. This option is also recommended if your event data store collects AWS Config configuration items, Audit Manager evidence, and events from outside of AWS.

For the first 366 days (the default retention period), storage is included at no additional cost with ingestion pricing. After 366 days, extended retention is available at pay-as-you-go pricing.

This is the default option.

Default retention period: 366 days

Maximum retention period: 3,653 days

Seven-year retention pricing

FIXED_RETENTION_PRICING

Recommended if expect to ingest more than 25 TB of event data per month and need a retention period of up to 7 years.

Retention is included with ingestion pricing at no additional charge.

Default retention period: 2,557 days

Maximum retention period: 2,557 days

Understanding CloudTrail Lake charges

The following tables provides information about how CloudTrail Lake event data stores and queries incur charges. For information about CloudTrail pricing, see AWS CloudTrail Pricing.

Charge type How you incur charges

Data ingestion (uncompressed data)

For CloudTrail Lake, you pay based on the uncompressed data ingested. The pricing option for the event data store determines the cost of ingesting events:

  • One-year extendable retention pricing: Offers ingestion pricing based on event type.

  • Seven-year retention pricing: Offers ingestion pricing based on the volume of data ingested. The greatest savings are achieved when the volume of data ingested monthly exceeds 25 TB.

Copying trail events

When you copy trail events to CloudTrail Lake, CloudTrail unzips the logs that are stored in gzip (compressed) format. Then CloudTrail copies the events contained in the logs to your event data store. The size of the uncompressed data could be greater than the actual Amazon S3 storage size. To get a general estimate of the size of the uncompressed data, multiply the size of the logs in the S3 bucket by 10.

Note

CloudTrail will not copy an event if its event time is older than the specified retention period. To determine the appropriate retention period, take the sum of the oldest event you want to copy in days and the number of days you want to retain the events in the event data store as demonstrated in this equation:

Retention period = oldest-event-in-days + number-days-to-retain

For example, if the oldest event you're copying is 45 days old and you want to keep the events in the event data store for a further 45 days, you would set the retention period to 90 days.

Data retention (optimized and compressed data)

CloudTrail Lake converts existing events in row-based JSON format to Apache ORC format. ORC is a columnar storage format that is optimized for fast retrieval of compressed data.

An event data store’s retention period determines how long event data is kept in the event data store. CloudTrail Lake determines whether to retain an event by checking if an event's event time is within the specified retention period. For example, if you specify a retention period of 90 days, CloudTrail will remove events when their event time is older than 90 days.

For event data stores using the Seven-year retention pricing option, storage is included with ingestion pricing at no additional charge.

For event data stores using the One-year extendable retention pricing option, storage is included at no charge with ingestion pricing for the first 366 days (the default retention period). After 366 days, storage is offered at pay-as-you-pricing and is charged based on the optimized and compressed data in the event data store.

Running queries in CloudTrail Lake (optimized and compressed data)

When you run queries in CloudTrail Lake, you pay based on the amount of optimized and compressed data scanned.

Recommendations for how you can reduce costs

This section provides recommendations for how you can reduce costs when working with CloudTrail Lake.

Choose a pricing option based on the type of events your event data store will collect and your expected monthly ingestion

When creating an event data store, choose a pricing option based on the type of events your event data store will collect and your expected monthly ingestion.

If you expect to ingest less than 25 TB of event data on a monthly basis and want a flexible retention period of up to 10 years, choose the One-year extendable retention pricing option. We also generally recommend this option for event data stores that collect AWS Config configuration items, Audit Manager evidence, and events from outside of AWS.

If you expect to ingest more than 25 TB of event data on a monthly basis and need a 7-year retention period, choose the Seven-year retention pricing option.

Evaluate your event data store's monthly ingestion over time

Evaluate the historical monthly ingestion of your event data store to see if there's a pricing option better suited to your needs.

If you have an existing event data store that uses the Seven-year retention pricing option and you ingest less than 25 TB of data on a monthly basis, consider updating the event data store to use One-year extendable retention pricing. For event data stores using the Seven-year retention pricing option, you can change the pricing option using the CloudTrail console, AWS CLI, or UpdateEventDataStore API operation.

If you have an existing event data store that uses the One-year extendable retention pricing option and you ingest more than 25 TB of event data on a monthly basis, consider whether Seven-year retention pricing would better suit your needs. To use the new pricing option, stop ingestion on your event data store and create a new event data store with the Seven-year retention pricing option.

Use advanced event selectors to filter out events that aren't of interest

When configuring an event data store for CloudTrail management or data events, filter out events that aren't of interest by using advanced event selectors.

If you're creating an event data store to collect management events, you can filter out AWS Key Management Service (AWS KMS) or Amazon Relational Database Service (Amazon RDS ) Data API events. Typically, AWS KMS actions such as Encrypt, Decrypt, and GenerateDataKey generate more than 99 percent of events.

If you're creating an event data store to collect data events, you can use advanced event selectors to filter on the eventName, resources.type, resources.ARN, and readOnly fields. For an example, see Tutorial: Create an event data store for S3 data events.

Choose a narrower time range when copying trail events

When copying trail events to CloudTrail Lake, specify a narrower start event time and end event time to reduce the amount of data ingested.

If you are copying trail events to CloudTrail Lake for historical analysis and do not want to ingest future events, deselect the option to ingest events so that you do not incur charges on ingesting any additional events.

Format queries to use a starting and ending eventTime

When you run queries in Lake, you pay based upon the amount of data scanned. You can constrain costs by specifying a starting and ending eventTime for the query.

Tools to help manage costs

AWS Budgets, a feature of AWS Billing and Cost Management, lets you set custom budgets that alert you when your costs or usage exceed (or are forecasted to exceed) your budgeted amount.

As you create event data stores, creating a budget for CloudTrail by using AWS Budgets is a recommended best practice, and can help you track your CloudTrail spending. Cost-based budgets help promote awareness of how much you might be billed for your CloudTrail use. Budget alerts notify you when your bill reaches a threshold that you define. When you receive a budget alert, you can make changes before the end of the billing cycle to manage your costs.

After you create a budget, you can use AWS Cost Explorer to see how your CloudTrail costs are influencing your overall AWS bill. In AWS Cost Explorer, after adding CloudTrail to the Service filter, you can compare your historical CloudTrail spending to that of your current month-to-date (MTD) spending, by both Region and account. This feature helps you monitor and detect unexpected costs in your monthly CloudTrail spending. Additional features in Cost Explorer let you compare CloudTrail spending to monthly spending at the specific resource level, providing information about what might be driving cost increases or decreases in your bill.

To get started with AWS Budgets, open AWS Billing and Cost Management, and then choose Budgets in the left navigation bar. We recommend configuring budget alerts as you create a budget to track CloudTrail spending. For more information about how to use AWS Budgets, see Managing Your Costs with Budgets and Best Practices for AWS Budgets.

Creating user-defined cost allocation tags for CloudTrail Lake event data stores

You can create user-defined cost allocation tags to track the query and ingestion costs for your CloudTrail Lake event data stores. A user-defined cost allocation tag is a key-value pair that you can associate with an event data store. After you activate cost allocation tags, AWS uses the tags to organize your resource costs on your cost allocation report.

For more information about activating tags, see Activating user-defined cost allocation tags.

See also