Logging Insights events - AWS CloudTrail

Logging Insights events

AWS CloudTrail Insights helps AWS users identify and respond to unusual activity associated with API calls and API error rates by continuously analyzing CloudTrail management events. CloudTrail Insights analyzes your normal patterns of API call volume and API error rates, also called the baseline, and generates Insights events when the call volume or error rates are outside normal patterns. Insights events on API call volume are generated for write management APIs, and Insights events on API error rate are generated for both read and write management APIs.

Note

To log Insights events on API call volume, the trail or event data store must log write management events. To log Insights events on API error rate, the trail or event data store must log read or write management events.

CloudTrail Insights analyzes management events that occur in a single Region, not globally. A CloudTrail Insights event is generated in the same Region as its supporting management events are generated.

Additional charges apply for Insights events. You will be charged separately if you enable Insights for both trails and event data stores. For more information, see AWS CloudTrail Pricing.

Understanding Insights events delivery

Unlike other types of events that CloudTrail captures, Insights events are logged only when CloudTrail detects changes in your account's API usage that differ significantly from the account's typical usage patterns.

Where CloudTrail delivers events and how long it takes to receive Insights events differs between trails and event data stores.

Insights events delivery for trails

If you've enabled Insights events on a trail and CloudTrail detects unusual activity, CloudTrail delivers Insights events to the /CloudTrail-Insight folder in the chosen destination S3 bucket for your trail. After you enable CloudTrail Insights for the first time on a trail, it can take up to 36 hours for CloudTrail to deliver the first Insights event, if unusual activity is detected.

If you turn off Insights events logging on a trail and then re-enable Insights events, or stop and restart logging on a trail, it can take up to 36 hours for CloudTrail to restart delivery of Insights events, if unusual activity is detected.

Insights events delivery for event data stores

If you've enabled Insights events on a source event data store, CloudTrail delivers Insights events to the destination event data store. After you enable CloudTrail Insights for the first time on the source event data store, it can take up to 7 days for CloudTrail to deliver the first Insights event to the destination event data store, if unusual activity is detected.

If you turn off Insights events logging on a source event data store and then re-enable Insights events, or stop and restart event ingestion on a source event data store, it can take up to 7 days for CloudTrail to restart delivery of Insights events, if unusual activity is detected. Additional charges apply for ingesting Insights events in CloudTrail Lake. You will be charged separately if you enable Insights for both trails and event data stores. For information about CloudTrail pricing, see AWS CloudTrail Pricing.

Logging Insights events with the AWS Management Console

You can enable Insights events on a trail or event data store using the console.

Enabling CloudTrail Insights events on an existing trail

Use the following procedure to enable CloudTrail Insights events on an existing trail. By default, Insights events are not enabled.

  1. In the left navigation pane of the CloudTrail console, open the Trails page, and choose a trail name.

  2. In Insights events choose Edit.

    Note

    Additional charges apply for logging Insights events. For CloudTrail pricing, see AWS CloudTrail Pricing.

  3. In Event type, choose Insights events.

  4. In Insights events, under Choose Insights types, choose API call rate, API error rate, or both. Your trail must be logging Write management events to log Insights events for API call rate. Your trail must be logging Read or Write management events to log Insights events for API error rate.

  5. Choose Save changes to save your changes.

It can take up to 36 hours for CloudTrail to deliver the first Insights events, if unusual activity is detected.

Enabling CloudTrail Insights events on an existing event data store

Use the following procedure to enable CloudTrail Insights events on an existing event data store. By default, Insights events are not enabled.

Additional charges apply for ingesting Insights events in CloudTrail Lake. You will be charged separately if you enable Insights for both trails and event data stores. For information about CloudTrail pricing, see AWS CloudTrail Pricing.

Note

You can only enable CloudTrail Insights events on event data stores containing CloudTrail management events. You cannot enable CloudTrail Insights events on other event data store types.

  1. In the left navigation pane of the CloudTrail console, under Lake, choose Event data stores.

  2. Choose the event data store name.

  3. In Management events, choose Edit.

  4. Choose Enable Insights.

  5. Choose the destination event data store where CloudTrail will deliver Insights events. The destination event data store will collect Insights events based upon the management event activity in this event data store. For information about how to create the destination event data store, see To create a destination event data store that logs Insights events.

  6. Under Choose Insights types, choose API call rate, API error rate, or both. Your event data store must be logging Write management events to log Insights events for API call rate. Your event data store must be logging Read or Write management events to log Insights events for API error rate.

  7. Choose Save changes to save your changes.

It can take up to 7 days for CloudTrail to deliver the first Insights events, if unusual activity is detected.

Logging Insights events with the AWS Command Line Interface

You can configure your trails and event data stores to log Insights events using the AWS CLI.

Note

To log Insights events on API call volume, the trail or event data store must log write management events. To log Insights events on API error rate, the trail or event data store must log read or write management events.

Logging Insights events for a trail using the AWS CLI

To view whether your trail is logging Insights events, run the get-insight-selectors command.

aws cloudtrail get-insight-selectors --trail-name TrailName

The following result shows the default settings for a trail. By default, trails don't log Insights events. The InsightType attribute value is empty, and no Insight event selectors are specified, because Insights event collection is not enabled.

If you do not add Insights selectors, the get-insight-selectors command returns the following error message: "An error occurred (InsightNotEnabledException) when calling the GetInsightSelectors operation: Trail name does not have Insights enabled. Edit the trail settings to enable Insights, and then try the operation again."

{ "InsightSelectors": [ ], "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/TrailName" }

To configure your trail to log Insights events, run the put-insight-selectors command. The following example shows how to configure your trail to include Insights events. Insights selector values can be ApiCallRateInsight, ApiErrorRateInsight, or both.

aws cloudtrail put-insight-selectors --trail-name TrailName --insight-selectors '[{"InsightType": "ApiCallRateInsight"},{"InsightType": "ApiErrorRateInsight"}]'

The following result shows the Insights event selector that is configured for the trail.

{ "InsightSelectors": [ { "InsightType": "ApiErrorRateInsight" }, { "InsightType": "ApiCallRateInsight" } ], "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/TrailName" }

Logging Insights events for an event data store using the AWS CLI

To enable Insights on an event data store, you must have a source event data store that logs management events and a destination event data store that logs Insights events.

To view whether Insights events are enabled on an event data store, run the get-insight-selectors command.

aws cloudtrail get-insight-selectors --event-data-store arn:aws:cloudtrail:us-east-1:123456789012:eventdatastore/EXAMPLE-f852-4e8f-8bd1-bcf6cEXAMPLE

To view whether an event data store is configured to receive Insights events or management events, run the get-event-data-store command.

aws cloudtrail get-event-data-store --event-data-store arn:aws:cloudtrail:us-east-1:123456789012:eventdatastore/EXAMPLE-d483-5c7d-4ac2-adb5dEXAMPLE

The following procedure shows you how to create the destination and source event data stores and then enable Insights events.

  1. Run the aws cloudtrail create-event-data-store command to create a destination event data store that collects Insights events. The value for eventCategory must be Insight. Replace retention-period-days with the number of days you would like to retain events in your event data store.

    If you are signed in with the management account for an AWS Organizations organization, include the --organization-enabled parameter if you want to give your delegated administrator access to the event data store.

    aws cloudtrail create-event-data-store \ --name insights-event-data-store \ --no-multi-region-enabled \ --retention-period retention-period-days \ --advanced-event-selectors '[ { "Name": "Select Insights events", "FieldSelectors": [ { "Field": "eventCategory", "Equals": ["Insight"] } ] } ]'

    The following is an example response.

    { "Name": "insights-event-data-store", "ARN": "arn:aws:cloudtrail:us-east-1:111122223333:eventdatastore/EXAMPLEf852-4e8f-8bd1-bcf6cEXAMPLE", "AdvancedEventSelectors": [ { "Name": "Select Insights events", "FieldSelectors": [ { "Field": "eventCategory", "Equals": [ "Insight" ] } ] } ], "MultiRegionEnabled": false, "OrganizationEnabled": false, "BillingMode": "EXTENDABLE_RETENTION_PRICING", "RetentionPeriod": "90", "TerminationProtectionEnabled": true, "CreatedTimestamp": "2023-11-08T15:22:33.578000+00:00", "UpdatedTimestamp": "2023-11-08T15:22:33.714000+00:00" }

    You will use the ARN (or ID suffix of the ARN) from the response as the value for the --insights-destination parameter in step 3.

  2. Run the aws cloudtrail create-event-data-store command to create a source event data store that logs management events. By default, event data stores log all management events. You don't need to specify the advanced event selectors if you want to log all management events. Replace retention-period-days with the number of days you would like to retain events in your event data store. If you are creating an organization event data store, include the --organization-enabled parameter.

    aws cloudtrail create-event-data-store --name source-event-data-store --retention-period retention-period-days

    The following is an example response.

    { "EventDataStoreArn": "arn:aws:cloudtrail:us-east-1:111122223333:eventdatastore/EXAMPLE9952-4ab9-49c0-b788-f4f3EXAMPLE", "Name": "source-event-data-store", "Status": "CREATED", "AdvancedEventSelectors": [ { "Name": "Default management events", "FieldSelectors": [ { "Field": "eventCategory", "Equals": [ "Management" ] } ] } ], "MultiRegionEnabled": true, "OrganizationEnabled": false, "BillingMode": "EXTENDABLE_RETENTION_PRICING", "RetentionPeriod": 90, "TerminationProtectionEnabled": true, "CreatedTimestamp": "2023-11-08T15:25:35.578000+00:00", "UpdatedTimestamp": "2023-11-08T15:25:35.714000+00:00" }

    You will use the ARN (or ID suffix of the ARN) from the response as the value for the --event-data-store parameter in step 3.

  3. Run the put-insight-selectors command to enable Insights events. Insights selector values can be ApiCallRateInsight, ApiErrorRateInsight, or both. For the --event-data-store parameter, specify the ARN (or ID suffix of the ARN) of the source event data store that logs management events and will enable Insights. For the --insights-destination parameter, specify the ARN (or ID suffix of the ARN) of the destination event data store that will log Insights events.

    aws cloudtrail put-insight-selectors --event-data-store arn:aws:cloudtrail:us-east-1:111122223333:eventdatastore/EXAMPLE9952-4ab9-49c0-b788-f4f3EXAMPLE --insights-destination arn:aws:cloudtrail:us-east-1:111122223333:eventdatastore/EXAMPLEf852-4e8f-8bd1-bcf6cEXAMPLE --insight-selectors '[{"InsightType": "ApiCallRateInsight"},{"InsightType": "ApiErrorRateInsight"}]'

    The following result shows the Insights event selector that is configured for the event data store.

    { "EventDataStoreARN": "arn:aws:cloudtrail:us-east-1:111122223333:eventdatastore/EXAMPLE9952-4ab9-49c0-b788-f4f3EXAMPLE", "InsightsDestination": "arn:aws:cloudtrail:us-east-1:111122223333:eventdatastore/EXAMPLEf852-4e8f-8bd1-bcf6cEXAMPLE", "InsightSelectors": [ { "InsightType": "ApiErrorRateInsight" }, { "InsightType": "ApiCallRateInsight" } ] }

    After you enable CloudTrail Insights for the first time on an event data store, it can take up to 7 days for CloudTrail to deliver the first Insights event, if unusual activity is detected.

    CloudTrail Insights analyzes management events that occur in a single Region, not globally. A CloudTrail Insights event is generated in the same Region as its supporting management events are generated.

    For an organization event data store, CloudTrail analyzes management events from each member's account instead of analyzing the aggregation of all management events for the organization.

Additional charges apply for ingesting Insights events in CloudTrail Lake. You will be charged separately if you enable Insights for both trails and event data stores. For information about CloudTrail pricing, see AWS CloudTrail Pricing.

Logging events with the AWS SDKs

Run the GetInsightSelectors operation to see whether your trail or event data store enables Insights events. You can configure your trails or event data stores to enable Insights events with the PutInsightSelectors operation. For more information, see the AWS CloudTrail API Reference.

Additional information for trails

This section provides additional information that is specific to trails. This section describes how you can view events for your subscribed trails from the Insights page in the CloudTrail console and how you can optionally send these events to CloudWatch Logs for monitoring.

Viewing Insights events for trails in the console

For trails, you can also access and view Insights events on the Insights page in the CloudTrail console. For more information about how to access and view Insights events in the console and by using the AWS CLI, see Viewing CloudTrail Insights events for trails in this guide.

The following image shows an example of Insights events for a trail. You open details pages for an Insights event by choosing an Insights event name from the Dashboard or Insights pages.

If you disable CloudTrail Insights on a trail, or stop logging on a trail (which disables CloudTrail Insights), you may have Insights events stored in your destination S3 bucket, or shown on the Insights page of the console, that date from the earlier time that you had Insights enabled.

Filter column

The left column lists Insights events that are related to the subject API, and that have the same Insights event type. The column lets you choose the Insights event about which you want more information. When you choose an event in this column, the event is highlighted in the graph on the Insights graph tab. By default, CloudTrail applies a filter that limits events shown on the CloudTrail events tab to those about the specific API that was called during the period of unusual activity that triggered the Insights event. To show all CloudTrail events called during the period of unusual activity, including events unrelated to the Insights event, turn off the filter.

Insights graph tab

On the Insights graph tab, the details page for an Insights event shows a graph of an API's call volume or error rate that occurred over a period of time before and after one or more Insights events are logged. In the graph, Insights events are highlighted with vertical bars, with the width of the bar showing the start and end time of the Insights event.

In this example, a vertical highlighting band shows unusual numbers of AWS Systems Manager SendCommand API calls in an account. In the highlighted area, because the number of SendCommand calls rose above the account's baseline average of 0.0442 calls per minute, CloudTrail logged an Insights event when it detected the unusual activity. The Insights event recorded that as many as 15 SendCommand calls were made in a five-minute period between 5:50 and 5:55 a.m. This is about two more calls to that API per minute than is expected for the account. In this example, the graph's time span is three hours: 4:30 a.m. PDT on July 15, 2021 to 7:30 a.m. PDT on July 15, 2021. This event has a start time of 6:00 a.m. PDT on July 15, 2021, and an end time two minutes later. An ending Insights event, not highlighted, shows that the unusual activity ended at about 6:16 a.m.

The baseline is calculated over the seven days preceding the start of an Insights event. Though the value of the baseline duration—the period that CloudTrail analyzes for normal activity on APIs—is approximately seven days, CloudTrail rounds the baseline duration to a whole integer day, so the exact baseline duration can vary.


                    A CloudTrail Insights detail page showing unusual API activity that was logged as an
                        Insights event.

You can use the Zoom command on the toolbar to zoom in on the ending Insights event, showing the start and end time. In this example, choosing Zoom, then dragging the Zoom cursor a very short distance over one edge of the highlighted Insights event expands the Insights event and shows more timeline detail.


                    A CloudTrail Insights event, zoomed in to show timeline detail.

To view CloudTrail events that were analyzed to determine unusual activity, open the CloudTrail events tab. In this example, CloudTrail analyzed 12 events, four of which triggered the Insights event.


                    A CloudTrail Insights event, zoomed in to show timeline detail.

The following image shows an Insights graph tab for an API error rate Insights event. The highlighted area shows that an Insights event was logged because occurrences of the NoSuchEntityException error on the GetRolePolicy IAM API call rose above the baseline average of 0.0017 NoSuchEntityException errors per minute on this API call, averaging 18 errors per minute during the insight period. The number of CloudTrail events that triggered the Insights event matches the Insights average of 18 NoSuchEntityException errors in one minute, in this example. Unlike an API call rate graph, the API error rate shows two lines, in contrasting colors: a line measuring calls to the IAM API, GetRolePolicy, that resulted in an unusual number of errors, and a line measuring the error on which unusual activity was logged, NoSuchEntityException.


                    A CloudTrail Insights detail page showing unusual error rate activity that was logged
                        as an Insights event.

Attributions tab

The Attributions tab shows the following information about an Insights event. Information on the Attributions tab can help you identify the causes and sources of Insights activity. Expand the top baseline areas to compare user identity, user agent, and error code activity during normal periods with those attributed during the Insights activity. In Top baseline user identity ARNs, Top baseline user agents, and Top baseline error codes, only the baseline average—the historic average of events for the API that are logged by the user identity, user agent, or that result in the error code, in approximately the seven days before the Insights event start time—is shown.


                    A CloudTrail Insights event detail page showing attributions.

The Attributions tab shows only top user identity ARNs and top user agents for an error rate Insights event, as shown in the following image. Top error codes are not necessary for error rate Insights events.


                    A CloudTrail Insights event detail page showing attributions for an error rate
                        Insights event.
  • Top user identity ARNs - This table shows up to the top five AWS users or IAM roles (user identities) that contributed to API calls during the unusual activity and baseline periods, in descending order by the average number of API calls contributed. The percentage of the averages as a total of activity that contributed to the unusual activity is shown in parentheses. If more than five user identity ARNs contributed to the unusual activity, their activity is summed up in an Other row.

  • Top user agents - This table shows up to the top five AWS tools by which the user identity contributed to API calls during the unusual activity and baseline periods, in descending order by the average number of API calls contributed. These tools include the AWS Management Console, AWS CLI, or the AWS SDKs. For example, a user agent named ec2.amazonaws.com indicates that the Amazon EC2 console was among the tools used to call the API. The percentage of the averages as a total of activity that contributed to the unusual activity is shown in parentheses. If more than five user agents contributed to the unusual activity, their activity is summed up in an Other row.

  • Top error codes - Only shown for API call rate Insights events. This table shows up to the top five error codes that occurred on API calls during the unusual activity and baseline periods, in descending order from largest number of API calls to smallest. The percentage of the averages as a total of activity that contributed to the unusual activity is shown in parentheses. If more than five error codes occurred during the unusual or baseline activity, their activity is summed up in an Other row.

    A value of None as one of the top five error code values means that a significant percentage of the calls that contributed to the Insights event did not result in errors. If the error code value is None, and there are no other error codes in the table, the values in the Insight average and Baseline average columns are the same as those for the Insights event overall. You can also see those values displayed in the Insight average and Baseline average legend on the Insights graph tab, under API calls per minute.

Baseline average and Insights average

Baseline average and Insights average are shown for top user identities, top user agents, and top error codes.

  • Baseline average - The typical rate of occurrences per minute on the API on which the Insights event was logged, as measured within approximately the preceding seven days, in a specific Region in your account.

  • Insights average - The rate of calls to or errors on this API that triggered the Insights event. The CloudTrail Insights average for the start event is the rate of calls or errors per minute on the API that triggered the Insights event. Typically, this is the first minute of unusual activity. The Insights average for the end event is the rate of API calls or errors per minute over the duration of the unusual activity, between the start Insights event and the end Insights event.

CloudTrail events tab

On the CloudTrail events tab, view related events that CloudTrail analyzed to determine that unusual activity occurred. By default, a filter is already applied for the Insights event name, which is also the name of the related API. To show all CloudTrail events logged during the period of unusual activity, turn off Only show events for selected Insights event. The CloudTrail events tab shows CloudTrail management events related to the subject API that occurred between the start and end time of the Insights event. These events help you perform deeper analysis to determine the probable cause of an Insights event, and reasons for unusual API and error rate activity.

Insights event record tab

Like any CloudTrail event, a CloudTrail Insights event is a record in JSON format. The Insights event record tab shows the JSON structure and content of the Insights start and end events, sometimes called the event payload. For more information about the fields and content of the Insights event record, see Record fields for Insights events and CloudTrail Insights insightDetails element in this guide.

Sending trail events to Amazon CloudWatch Logs

CloudTrail supports sending Insights events for trails to CloudWatch Logs. When you configure your trail to send Insights events to your CloudWatch Logs log group, CloudTrail Insights sends only the events that you specify in your trail. For example, if you configure your trail to log management and Insights events, your trail delivers management and Insights events to your CloudWatch Logs log group. For more information, see Monitoring CloudTrail Log Files with Amazon CloudWatch Logs.