Monitor AWS Secrets Manager secrets scheduled for deletion by using Amazon CloudWatch
You can use a combination of AWS CloudTrail, Amazon CloudWatch Logs, and Amazon Simple Notification Service (Amazon SNS) to create an alarm that notifies you of any attempts to access a secret pending deletion. If you receive a notification from an alarm, you might want to cancel deletion of the secret to give yourself more time to determine if you really want to delete it. Your investigation might result in the secret being restored because you still need the secret. Alternatively, you might need to update the user with details of the new secret to use.
The following procedures explain how to receive a notification when a request for the
GetSecretValue
operation that results in a specific error message written
to your CloudTrail log files. Other API operations can be performed on the secret without
triggering the alarm. This CloudWatch alarm detects usage that might indicate a person or
application using outdated credentials.
Before you begin these procedures, you must turn on CloudTrail in the AWS Region and account where you intend to monitor AWS Secrets ManagerAPI requests. For instructions, go to Creating a trail for the first time in the AWS CloudTrail User Guide.
Step 1: Configure CloudTrail log file delivery to CloudWatch logs
You must configure delivery of your CloudTrail log files to CloudWatch Logs. You do this so CloudWatch Logs can monitor them for Secrets Manager API requests to retrieve a secret pending deletion.
To configure CloudTrail log file delivery to CloudWatch Logs
-
Open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/
. -
On the top navigation bar, choose the AWS Region to monitor secrets.
-
In the left navigation pane, choose Trails, and then choose the name of the trail to configure for CloudWatch.
-
On the Trails Configuration page, scroll down to the CloudWatch Logs section, and then choose the edit icon (
).
-
For New or existing log group, type a name for the log group, such as
CloudTrail/MyCloudWatchLogGroup
. -
For IAM role, you can use the default role named CloudTrail_CloudWatchLogs_Role. This role has a default role policy with the required permissions to deliver CloudTrail events to the log group.
-
Choose Continue to save your configuration.
-
On the AWS CloudTrail will deliver CloudTrail events associated with API activity in your account to your CloudWatch Logs log group page, choose Allow.
Step 2: Create the CloudWatch alarm
To receive a notification when a Secrets Manager GetSecretValue
API operation
requests to access a secret pending deletion, you must create a CloudWatch alarm and configure
notification.
To create a CloudWatch alarm
-
Sign in to the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
. -
On the top navigation bar, choose the AWS Region where you want to monitor secrets.
-
In the left navigation pane, choose Logs.
-
In the list of Log Groups, select the check box next to the log group you created in the previous procedure, such as CloudTrail/MyCloudWatchLogGroup. Then choose Create Metric Filter.
-
For Filter Pattern, type or paste the following:
{ $.eventName = "GetSecretValue" && $.errorMessage = "*secret because it was marked for deletion*" }
Choose Assign Metric.
-
On the Create Metric Filter and Assign a Metric page, do the following:
-
For Metric Namespace, type
CloudTrailLogMetrics
. -
For Metric Name, type
AttemptsToAccessDeletedSecrets
. -
Choose Show advanced metric settings, and then if necessary for Metric Value, type
1
. -
Choose Create Filter.
-
-
In the filter box, choose Create Alarm.
-
In the Create Alarm window, do the following:
-
For Name, type
AttemptsToAccessDeletedSecretsAlarm
. -
Whenever:, for is:, choose >=, and then type
1
. -
Next to Send notification to:, do one of the following:
-
To create and use a new Amazon SNS topic, choose New list, and then type a new topic name. For Email list:, type at least one email address. You can type more than one email address by separating them with commas.
-
To use an existing Amazon SNS topic, choose the name of the topic to use. If a list doesn't exist, choose Select list.
-
-
Choose Create Alarm.
-
Step 3: Test the CloudWatch alarm
To test your alarm, create a secret and then schedule it for deletion. Then, try to retrieve the secret value. You shortly receive an email at the address you configured in the alarm. It alerts you to the use of a secret scheduled for deletion.