Limitations Prerequisites To create an event data store for configuration items Configuration item schema

Create an event data store for configuration items with the console

You can create an event data store to include AWS Config configuration items, and use the event data store to investigate non-compliant changes to your production environments. With an event data store, you can relate non-compliant rules to the users and resources associated with the changes. A configuration item represents a point-in-time view of the attributes of a supported AWS resource that exists in your account. AWS Config creates a configuration item whenever it detects a change to a resource type that it is recording. AWS Config also creates configuration items when a configuration snapshot is captured.

You can use both AWS Config and CloudTrail Lake to run queries against your configuration items. You can use AWS Config to query the current configuration state of AWS resources based on configuration properties for a single AWS account and AWS Region, or across multiple accounts and Regions. In contrast, you can use CloudTrail Lake to query across diverse data sources such as CloudTrail events, configuration items, and rule evaluations. CloudTrail Lake queries cover all AWS Config configuration items including resource configuration and compliance history.

Creating an event data store for configuration items doesn't impact existing AWS Config advanced queries, or any configured AWS Config aggregators. You can continue to run advanced queries using AWS Config, and AWS Config continues to deliver history files to your S3 buckets.

CloudTrail Lake event data stores incur charges. When you create an event data store, you choose the pricing option you want to use for the event data store. The pricing option determines the cost for ingesting and storing events, and the default and maximum retention period for the event data store. For information about CloudTrail pricing and managing Lake costs, see AWS CloudTrail Pricing and Managing CloudTrail Lake costs.

Limitations

The following limitations apply to event data stores for configuration items.

No support for custom configuration items
No support for event filtering using advanced event selectors

Prerequisites

Before you create your event data store, set up AWS Config recording for all your accounts and Regions. You can use Quick Setup, a capability of AWS Systems Manager, to quickly create a configuration recorder powered by AWS Config.

Note

You are charged service usage fees when AWS Config starts recording configurations. For more information about pricing, see AWS Config Pricing. For information about managing the configuration recorder, see Managing the Configuration Recorder in the AWS Config Developer Guide.

Additionally, the following actions are recommended, but are not required to create an event data store.

Set up an Amazon S3 bucket to receive a configuration snapshot on request and configuration history. For more information about snapshots, see Managing the Delivery Channel and Delivering Configuration Snapshot to an Amazon S3 Bucket in the AWS Config Developer Guide.
Specify the rules that you want AWS Config to use to evaluate compliance information for the recorded resource types. Several of the CloudTrail Lake sample queries for AWS Config require AWS Config Rules to evaluate the compliance state of your AWS resources. For more information about AWS Config Rules, see Evaluating Resources with AWS Config Rules in the AWS Config Developer Guide.

To create an event data store for configuration items

Sign in to the AWS Management Console and open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.
From the navigation pane, under Lake, choose Event data stores.
Choose Create event data store.
On the Configure event data store page, in General details, enter a name for the event data store. A name is required.
Choose the Pricing option that you want to use for your event data store. The pricing option determines the cost for ingesting and storing events, and the default and maximum retention periods for your event data store. For more information, see AWS CloudTrail Pricing and Managing CloudTrail Lake costs.

The following are the available options:
- One-year extendable retention pricing - Generally recommended if you expect to ingest less than 25 TB of event data per month and want a flexible retention period of up to 10 years. For the first 366 days (the default retention period), storage is included at no additional charge with ingestion pricing. After 366 days, extended retention is available at pay-as-you-go pricing. This is the default option.
  - Default retention period: 366 days
  - Maximum retention period: 3,653 days
- Seven-year retention pricing - Recommended if you expect to ingest more than 25 TB of event data per month and need a retention period of up to 7 years. Retention is included with ingestion pricing at no additional charge.
  - Default retention period: 2,557 days
  - Maximum retention period: 2,557 days
Specify a retention period for the event data store. Retention periods can be between 7 days and 3,653 days (about 10 years) for the One-year extendable retention pricing option, or between 7 days and 2,557 days (about seven years) for the Seven-year retention pricing option.

CloudTrail Lake determines whether to retain an event by checking if the eventTime of the event is within the specified retention period. For example, if you specify a retention period of 90 days, CloudTrail will remove events when their eventTime is older than 90 days.
(Optional) To enable encryption using AWS Key Management Service, choose Use my own AWS KMS key. Choose New to have an AWS KMS key created for you, or choose Existing to use an existing KMS key. In Enter KMS alias, specify an alias, in the format alias/MyAliasName. Using your own KMS key requires that you edit your KMS key policy to allow CloudTrail logs to be encrypted and decrypted. For more information, see Configure AWS KMS key policies for CloudTrail. CloudTrail also supports AWS KMS multi-Region keys. For more information about multi-Region keys, see Using multi-Region keys in the AWS Key Management Service Developer Guide.

Using your own KMS key incurs AWS KMS costs for encryption and decryption. After you associate an event data store with a KMS key, the KMS key cannot be removed or changed.

Note
To enable AWS Key Management Service encryption for an organization event data store, you must use an existing KMS key for the management account.
(Optional) If you want to query against your event data using Amazon Athena, choose Enable in Lake query federation. Federation lets you view the metadata associated with the event data store in the AWS Glue Data Catalog and run SQL queries against the event data in Athena. The table metadata stored in the AWS Glue Data Catalog lets the Athena query engine know how to find, read, and process the data that you want to query. For more information, see Federate an event data store.

To enable Lake query federation, choose Enable and then do the following:
1. Choose whether you want to create a new role or use an existing IAM role. AWS Lake Formation uses this role to manage permissions for the federated event data store. When you create a new role using the CloudTrail console, CloudTrail automatically creates a role with the required permissions. If you choose an existing role, be sure the policy for the role provides the required minimum permissions.
2. If you are creating a new role, enter a name to identify the role.
3. If you are using an existing role, choose the role you want to use. The role must exist in your account.
(Optional) In the Tags section, you can add up to 50 tag key pairs to help you identify, sort, and control access to your event data store. For more information about how to use IAM policies to authorize access to an event data store based on tags, see Examples: Denying access to create or delete event data stores based on tags. For more information about how you can use tags in AWS, see Tagging your AWS resources in the Tagging AWS Resources User Guide.
Choose Next.
On the Choose events page, choose AWS events, and then choose Configuration items.
CloudTrail stores the event data store resource in the Region in which you create it, but by default, the configuration items collected in the data store are from all Regions in your account that have recording enabled. Optionally, you can select Include only the current region in my event data store to include only configuration items that are captured in the current Region. If you do not choose this option, your event data store includes configuration items from all Regions that have recording enabled.
To have your event data store collect configuration items from all accounts in an AWS Organizations organization, select Enable for all accounts in my organization. You must be signed in to the management account or delegated administrator account for the organization to create an event data store that collects configuration items for an organization.
Choose Next to review your choices.
On the Review and create page, review your choices. Choose Edit to make changes to a section. When you're ready to create the event data store, choose Create event data store.
The new event data store is visible in the Event data stores table on the Event data stores page.

From this point forward, the event data store captures configuration items. Configuration items that occurred before you created the event data store are not in the event data store.

Sample queries

You can now run queries on your new event data store. The Sample queries tab on the CloudTrail console provides example queries to get you started. The following are a few of the sample queries that you can run against your configuration item event data store.

Description	Query
Find which user performed an action that resulted in a non-compliant status by joining a configuration item event data store with a CloudTrail event data store.	SELECT element_at(config1.eventData.configuration, 'targetResourceId') as targetResourceId, element_at(config1.eventData.configuration, 'complianceType') as complianceType, config2.eventData.resourceType, cloudtrail.userIdentity FROM config_event_data_store_ID as config1 JOIN config_event_data_store_ID as config2 on element_at(config1.eventData.configuration, 'targetResourceId') = config2.eventData.resourceId JOIN cloudtrail_event_data_store_ID as cloudtrail on config2.eventData.arn = element_at(cloudtrail.resources, 1).arn WHERE element_at(config1.eventData.configuration, 'configRuleList') is not null AND element_at(config1.eventData.configuration, 'complianceType') = 'NON_COMPLIANT' AND cloudtrail.eventTime > '2022-11-14 00:00:00' AND config2.eventData.resourceType = 'AWS::DynamoDB::Table'
Find all AWS Config rules and return the compliance state from configuration items generated within the past day.	SELECT eventData.configuration, eventData.accountId, eventData.awsRegion, eventData.resourceName, eventData.resourceCreationTime, element_at(eventData.configuration,'complianceType') AS complianceType, element_at(eventData.configuration, 'configRuleList') AS configRuleList, element_at(eventData.configuration, 'resourceId') AS resourceId, element_at(eventData.configuration, 'resourceType') AS resourceType FROM config_event_data_store_ID WHERE eventData.resourceType = 'AWS::Config::ResourceCompliance' AND eventTime > '2022-11-22 00:00:00' ORDER BY eventData.resourceCreationTime DESC limit 10
Find the total count of AWS Config resources grouped by resource type, account ID, and Region.	`SELECT eventData.resourceType, eventData.awsRegion, eventData.accountId, COUNT (*) AS resourceCount FROM config_event_data_store_ID WHERE eventTime > '2022-11-22 00:00:00' GROUP BY eventData.resourceType, eventData.awsRegion, eventData.accountId`
Find the resource creation time for all AWS Config configuration items generated on a specific date.	`SELECT eventData.configuration, eventData.accountId, eventData.awsRegion, eventData.resourceId, eventData.resourceName, eventData.resourceType, eventData.availabilityZone, eventData.resourceCreationTime FROM config_event_data_store_ID WHERE eventTime > '2022-11-16 00:00:00' AND eventTime < '2022-11-17 00:00:00' ORDER BY eventData.resourceCreationTime DESC limit 10;`

For more information about creating and editing queries, see Create or edit a query with the CloudTrail console.

Configuration item schema

The following table describes the required and optional schema elements that match those in configuration item records. The contents of eventData are provided by your configuration items; other fields are provided by CloudTrail after ingestion.

CloudTrail event record contents are described in more detail in CloudTrail record contents.

Fields that are provided by CloudTrail after ingestion
Fields that are provided by your events

Fields that are provided by CloudTrail after ingestion
Field name	Input type	Requirement	Description
eventVersion	string	Required	The version of the AWS event format.
eventCategory	string	Required	The event category. For configuration items, the valid value is `ConfigurationItem`.
eventType	string	Required	The event type. For configuration items, the valid value is `AwsConfigurationItem`.
eventID	string	Required	A unique ID for an event.
eventTime	string	Required	The event timestamp, in `yyyy-MM-DDTHH:mm:ss` format, in Universal Coordinated Time (UTC).
awsRegion	string	Required	The AWS Region to which to assign an event.
recipientAccountId	string	Required	Represents the AWS account ID that received this event.
addendum	addendum	Optional	Shows information about why an event was delayed. If information was missing from an existing event, the addendum block includes the missing information and a reason for why it was missing.

Fields in `eventData` are provided by your configuration items
Field name	Input type	Requirement	Description
eventData	-	Required	Fields in eventData are provided by your configuration items.
configurationItemVersion	string	Optional	The version of the configuration item from its source.
configurationItemCaptureTime	string	Optional	The time when the configuration recording was initiated.
configurationItemStatus	string	Optional	The configuration item status. Valid values are `OK`, `ResourceDiscovered`, `ResourceNotRecorded`, `ResourceDeleted`, and `ResourceDeletedNotRecorded`.
accountId	string	Optional	The 12-digit AWS account ID associated with the resource.
resourceType	string	Optional	The type of AWS resource. For more information about valid resource types, see ConfigurationItem in the AWS Config API Reference.
resourceId	string	Optional	The ID of the resource (for example., sg-`xxxxxx`).
resourceName	string	Optional	The custom name of the resource, if available.
arn	string	Optional	Amazon Resource Name (ARN) associated with the resource.
awsRegion	string	Optional	The AWS Region where the resource resides.
availabilityZone	string	Optional	The Availability Zone associated with the resource.
resourceCreationTime	string	Optional	The time stamp when the resource was created.
configuration	JSON	Optional	The description of the resource configuration.
supplementaryConfiguration	JSON	Optional	Configuration attributes that AWS Config returns for certain resource types to supplement the information returned for the configuration parameter.
relatedEvents	string	Optional	A list of CloudTrail event IDs.
relationships	-	Optional	A list of related AWS resources.
name	string	Optional	The type of relationship with the related resource.
resourceType	string	Optional	The resource type of the related resource.
resourceId	string	Optional	The ID of the related resource (for example, sg-`xxxxxx`).
resourceName	string	Optional	The custom name of the related resource, if available.
tags	JSON	Optional	A mapping of key value tags associated with the resource.

The following example shows the hierarchy of schema elements that match those in configuration item records.


{
  "eventVersion": String,
  "eventCategory: String,
  "eventType": String,
  "eventID": String,
  "eventTime": String,
  "awsRegion": String,
  "recipientAccountId": String,
  "addendum": Addendum,
  "eventData": {
      "configurationItemVersion": String,
      "configurationItemCaptureTime": String,
      "configurationItemStatus": String,
      "configurationStateId": String,
      "accountId": String,
      "resourceType": String,
      "resourceId": String,
      "resourceName": String,
      "arn": String,
      "awsRegion": String, 
      "availabilityZone": String,
      "resourceCreationTime": String,
      "configuration": {
        JSON,
      },
      "supplementaryConfiguration": {
        JSON,
      },
      "relatedEvents": [
        String
      ],
      "relationships": [
        struct{
          "name" : String,
          "resourceType": String,
          "resourceId": String,
          "resourceName": String
        }
      ],
     "tags": {
       JSON
     }
    }
  }
}

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Create an event data store for Insights events

Create an event data store for events outside of AWS