Logging AWS Secrets Manager API Calls with AWS CloudTrail Amazon CloudWatch Events

Monitoring the use of your AWS Secrets Manager secrets

As a best practice, you should monitor your secrets to ensure usage of your secrets and log any changes to them. This helps you to ensure that any unexpected usage or change can be investigated, and unwanted changes can be rolled back. AWS Secrets Manager currently supports two AWS services that enable you to monitor your organization and activity.

Topics

Logging AWS Secrets Manager API Calls with AWS CloudTrail
Amazon CloudWatch Events

Logging AWS Secrets Manager API Calls with AWS CloudTrail

AWS Secrets Manager integrates with AWS CloudTrail, a service that provides a record of actions taken by a user, role or an AWS service in Secrets Manager. CloudTrail captures all API calls for Secrets Manager as events, including calls from the Secrets Manager console and from code calls to the Secrets Manager APIs. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for Secrets Manager. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in Event history. Using the information collected by CloudTrail, you can determine the request sent to Secrets Manager, the IP address of the request, who sent the request, when the time of the request, and additional details.

To learn more about CloudTrail see the AWS CloudTrail User Guide.

Logging AWS Secrets Manager non-API Events

In addition to logging AWS API calls, CloudTrail captures other related events that might have a security or compliance impact on your AWS account or might help you troubleshoot operational problems. CloudTrail records these events as non-API service events.

Secrets Manager has three non-API service events:

RotationAbandoned event - a mechanism to inform you that the Secrets Manager service removed the AWSPENDING label from an existing version of a secret. When you manually create a new version of a secret, you send a message signalling the abandonment of the current ongoing rotation in favor of the new secret version. As a result, Secrets Manager removes the AWSPENDING label to allow future rotations to succeed and publish a CloudTrail event to provide awareness of the change.
RotationStarted event - a mechanism that notifies you of a secret starting rotation.
RotationSucceeded event - a mechanism that notifies you of a successful rotation event.
RotationFailed event - a mechanism to inform you that secret rotation failed for an application.
StartSecretVersionDelete event - a mechanism that notifies you of the start deletion for a secret version.
CancelSecretVersionDelete event - a mechanism that notifies you of a delete cancellation for a secret version.
EndSecretVersionDelete event - a mechanism that notifies you of an ending secret version deletion.

Secrets Manager Information in CloudTrail

When you create your AWS account, AWS enables CloudTrail. When activity occurs in Secrets Manager, CloudTrail records the activity in a CloudTrail event along with other AWS service events in Event history. You can view, search, and download recent events in your AWS account. For more information, see Viewing Events with CloudTrail Event History.

For an ongoing record of events in your AWS account, including events for Secrets Manager, create a trail. A trail enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all regions. The trail logs events from all regions in the AWS partition and delivers the log files to the Amazon S3 bucket you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in CloudTrail logs. For more information, see:

CloudTrail logs all Secrets Manager actions and documents the actions in the AWS Secrets Manager API Reference. For example, calls to the CreateSecret, GetSecretValue and RotateSecret sections generate entries in the CloudTrail log files.

Every event or log entry contains information about who generated the request. The identity information helps you determine the following:

If root or IAM user credentials generated the request.
If temporary security credentials for a role or federated user generated the request.
If another AWS service generated the request.

For notification of log file delivery, you can configure CloudTrail to publish Amazon SNS notifications. For more information, see Configuring Amazon SNS Notifications for CloudTrail.

You also can aggregate AWS Secrets Manager log files from multiple AWS Regions and multiple AWS accounts into a single Amazon S3 bucket.

For more information, see Receiving CloudTrail Log Files from Multiple Regions and Receiving CloudTrail Log Files from Multiple Accounts.

Retrieving Secrets Manager Log File Entries

You can retrieve individual events from CloudTrail by using any of the following techniques:

To retrieve Secrets Manager events from CloudTrail logs

Using the AWS Management Console

The CloudTrail console enables you to view events that occurred within the past 90 days.

Open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.
Ensure that the console points to the region where your events occurred. The console shows only those events that occurred in the selected region. Choose the region from the drop-down list in the upper-right corner of the console.
In the left-hand navigation pane, choose Event history.
Choose Filter criteria and/or a Time range to help you find the event that you're looking for. For example, to see all Secrets Manager events, for Select attribute, choose Event source. Then, for Enter event source, choose secretsmanager.amazonaws.com.
To see additional details, choose the expand arrow next to event. To see all of the information available, choose View event.

Using the AWS CLI or SDK Operations

Open a command window to run AWS CLI commands.

Run a command similar to the following example. For readability here, the output displays as word-wrapped, but the real output doesn't.


$ aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=EventSource,AttributeValue=secretsmanager.amazonaws.com
{
    "Events": [
        {
            "EventId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
            "EventName": "CreateSecret",
            "EventTime": 1525106994.0,
            "Username": "Administrator",
            "Resources": [],
            "CloudTrailEvent": "{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AKIAIOSFODNN7EXAMPLE\",
                  \"arn\":\"arn:aws:iam::123456789012:user/Administrator\",\"accountId\":\"123456789012\",\"accessKeyId\":\"AKIAIOSFODNN7EXAMPLE\",
                  \"userName\":\"Administrator\"},\"eventTime\":\"2018-04-30T16:49:54Z\",\"eventSource\":\"secretsmanager.amazonaws.com\",
                  \"eventName\":\"CreateSecret\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"192.168.100.101\",
                  \"userAgent\":\"<useragent string>\",\"requestParameters\":{\"name\":\"MyTestSecret\",
                  \"clientRequestToken\":\"EXAMPLE2-90ab-cdef-fedc-ba987EXAMPLE\"},\"responseElements\":null,
                  \"requestID\":\"EXAMPLE3-90ab-cdef-fedc-ba987EXAMPLE\",\"eventID\":\"EXAMPLE4-90ab-cdef-fedc-ba987EXAMPLE\",
                  \"eventType\":\"AwsApiCall\",\"recipientAccountId\":\"123456789012\"}"
        }
    ]
}

Understanding Secrets Manager log file entries

A trail enables delivery of events as log files to a specified Amazon S3 bucket. CloudTrail log files contain one or more log entries. An event represents a single request from any source and includes information about the requested action, the date and time of the action, request parameters, and so on. CloudTrail log files does not collect an ordered stack trace of the public API calls, so they do not appear in any specific order.

The following example shows a CloudTrail log entry for a sample CreateSecret call:


  {
    "eventVersion": "1.05",
    "userIdentity": {
      "type": "Root",
      "principalId": "123456789012",
      "arn": "arn:aws:iam::123456789012:root",
      "accountId": "123456789012",
      "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
      "userName": "myusername",
      "sessionContext": {"attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2018-04-03T17:43:50Z"
      }}
    },
    "eventTime": "2018-04-03T17:50:55Z",
    "eventSource": "secretsmanager.amazonaws.com",
    "eventName": "CreateSecret",
    "awsRegion": "us-west-2",
    "requestParameters": {
      "name": "MyDatabaseSecret",
      "clientRequestToken": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE"
    },
    "responseElements": null,
    "requestID": "EXAMPLE2-90ab-cdef-fedc-ba987EXAMPLE",
    "eventID": "EXAMPLE3-90ab-cdef-fedc-ba987EXAMPLE",
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}

The following example shows a CloudTrail log entry for a sample DeleteSecret call:


{
    "eventVersion": "1.05",
    "userIdentity": {
      "type": "Root",
      "principalId": "123456789012",
      "arn": "arn:aws:iam::123456789012:root",
      "accountId": "123456789012",
      "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
      "userName": "myusername",
      "sessionContext": {"attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2018-04-03T17:43:50Z"
      }}
    },
    "eventTime": "2018-04-03T17:51:02Z",
    "eventSource": "secretsmanager.amazonaws.com",
    "eventName": "DeleteSecret",
    "awsRegion": "us-west-2",
    "requestParameters": {
      "recoveryWindowInDays": 30,
      "secretId": "MyDatabaseSecret"
    },
    "responseElements": {
      "name": "MyDatabaseSecret",
      "deletionDate": "May 3, 2018 5:51:02 PM",
      "aRN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyDatabaseSecret-a1b2c3"
    },
    "requestID": "EXAMPLE2-90ab-cdef-fedc-ba987EXAMPLE",
    "eventID": "EXAMPLE3-90ab-cdef-fedc-ba987EXAMPLE",
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}

Amazon CloudWatch Events

Secrets Manager can work with CloudWatch Events to trigger alerts when administrator-specified operations occur in an organization. For example, because of the sensitivity of such operations, administrators might want to be warned of deleted secrets or secret rotation. You might want an alert if anyone tries to use a secret version in the waiting period to be deleted. You can configure CloudWatch Events rules that look for these operations and then send the generated events to administrator defined "targets". A target could be an Amazon SNS topic that emails or text messages to subscribers. You can also create a simple AWS Lambda function triggered by the event, which logs the details of the operation for your later review.

To learn more about CloudWatch Events, including how to configure and enable it, see the Amazon CloudWatch Events User Guide.

Monitoring secret versions scheduled for deletion

You can use a combination of AWS CloudTrail, Amazon CloudWatch Logs, and Amazon Simple Notification Service (Amazon SNS) to create an alarm that notifies you of any attempts to access a version of a secret pending deletion. If you receive a notification from an alarm, you might want to cancel deletion of the secret to give yourself more time to determine if you really want to delete it. Your investigation might result in the secret being restored because you still need the secret. Alternatively, you might need to update the user with details of the new secret to use.

The following procedures explain how to receive a notification when a request for the GetSecretValue operation that results in a specific error message written to your CloudTrail log files. Other API operations can be performed on the version of the secret without triggering the alarm. This CloudWatch alarm detects usage that might indicate a person or application using outdated credentials.

Before you begin these procedures, you must turn on CloudTrail in the AWS Region and account where you intend to monitor AWS Secrets ManagerAPI requests. For instructions, go to Creating a trail for the first time in the AWS CloudTrail User Guide.

Steps

Part 1: Configuring CloudTrail log file delivery to CloudWatch logs
Part 2: Create the CloudWatch alarm
Part 3: Monitoring CloudWatch for Deleted Secrets

Part 1: Configuring CloudTrail log file delivery to CloudWatch logs

You must configure delivery of your CloudTrail log files to CloudWatch Logs. You do this so CloudWatch Logs can monitor them for Secrets Manager API requests to retrieve a version of a secret pending deletion.

To configure CloudTrail log file delivery to CloudWatch Logs

Open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.
On the top navigation bar, choose the AWS Region to monitor secrets.
In the left navigation pane, choose Trails, and then choose the name of the trail to configure for CloudWatch.
On the Trails Configuration page, scroll down to the CloudWatch Logs section, and then choose the edit icon ( ).
For New or existing log group, type a name for the log group, such as CloudTrail/MyCloudWatchLogGroup.
For IAM role, you can use the default role named CloudTrail_CloudWatchLogs_Role. This role has a default role policy with the required permissions to deliver CloudTrail events to the log group.
Choose Continue to save your configuration.
On the AWS CloudTrail will deliver CloudTrail events associated with API activity in your account to your CloudWatch Logs log group page, choose Allow.

Part 2: Create the CloudWatch alarm

To receive a notification when a Secrets Manager GetSecretValue API operation requests to access a version of a secret pending deletion, you must create a CloudWatch alarm and configure notification.

Creating a CloudWatch alarm to monitors usage of a version of a secret pending deletion

Sign in to the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
On the top navigation bar, choose the AWS Region where you want to monitor secrets.
In the left navigation pane, choose Logs.
In the list of Log Groups, select the check box next to the log group you created in the previous procedure, such as CloudTrail/MyCloudWatchLogGroup. Then choose Create Metric Filter.

For Filter Pattern, type or paste the following:


{ $.eventName = "GetSecretValue" && $.errorMessage = "*secret because it was marked for deletion*" }

Choose Assign Metric.

On the Create Metric Filter and Assign a Metric page, do the following:
1. For Metric Namespace, type CloudTrailLogMetrics.
2. For Metric Name, type AttemptsToAccessDeletedSecrets.
3. Choose Show advanced metric settings, and then if necessary for Metric Value, type 1.
4. Choose Create Filter.
In the filter box, choose Create Alarm.
In the Create Alarm window, do the following:
1. For Name, type AttemptsToAccessDeletedSecretsAlarm.
2. Whenever:, for is:, choose >=, and then type 1.
3. Next to Send notification to:, do one of the following:
  - To create and use a new Amazon SNS topic, choose New list, and then type a new topic name. For Email list:, type at least one email address. You can type more than one email address by separating them with commas.
  - To use an existing Amazon SNS topic, choose the name of the topic to use. If a list doesn't exist, choose Select list.
4. Choose Create Alarm.

Part 3: Monitoring CloudWatch for Deleted Secrets

You have created your alarm. To test it, create a version of a secret and then schedule it for deletion. Then, try to retrieve the secret value. You shortly receive an email at the address configured in the alarm. It alerts you to the use of a secret version scheduled for deletion.

Warning Javascript is disabled or is unavailable in your browser.

To use the AWS Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Using Secrets Manager with VPC Endpoints

Working with other services