Troubleshoot AWS Secrets Manager rotation of secrets - AWS Secrets Manager

Troubleshoot AWS Secrets Manager rotation of secrets

Use the information here to help you diagnose and fix common errors that you might encounter when you're rotating Secrets Manager secrets.

Rotating secrets in AWS Secrets Manager requires you to use a Lambda function that defines how to interact with the database or service that owns the secret.

I want to find the diagnostic logs for my Lambda rotation function

When the rotation function doesn't operate the way you expect, you should first check the CloudWatch logs. Secrets Manager provides template code for the Lambda rotation function, and this code writes error messages to the CloudWatch log.

To view the CloudWatch logs for your Lambda function

  1. Open the AWS Lambda console at

  2. From the list of functions, choose the name of the Lambda function associated with your secret.

  3. On the Monitor tab, choose Logs, and then choose View logs in CloudWatch.

    The CloudWatch console opens and displays the logs for your function.

I can't predict when rotation will start

You can predict only the date of the next rotation, not the time.

Secrets Manager schedules the next rotation when the previous one is complete. Secrets Manager schedules the date by adding the rotation interval (number of days) to the actual date of the last rotation. The service chooses the hour within that 24-hour date window randomly. The minute is also chosen somewhat randomly, but is weighted towards the top of the hour and influenced by a variety of factors that help distribute load.

I get "access denied" when trying to configure rotation for my secret

When you add a Lambda rotation function Amazon Resource Name (ARN) to your secret, Secrets Manager checks the permissions of the function. The role policy for the function must grant the Secrets Manager service principal permission to invoke the function (lambda:InvokeFunction).

You can add this permission by running the following AWS CLI command:

aws lambda add-permission --function-name ARN_of_lambda_function --principal --action lambda:InvokeFunction --statement-id SecretsManagerAccess

My first rotation fails after I enable rotation

When you enable rotation for a secret that uses a "master" secret to change the credentials on the secured service, Secrets Manager automatically configures most elements required for rotation. However, Secrets Manager can't automatically grant permission to read the master secret to your Lambda function. You must explicitly grant this permission yourself. Specifically, you grant the permission by adding it to the policy attached to the IAM role attached to your Lambda rotation function. That policy must include the following statement; this is only a statement, not a complete policy. For the complete policy, see the second sample policy in the section CloudTrail shows access-denied errors during rotation.

{ "Sid": "AllowAccessToMasterSecret", "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "ARN_of_master_secret" }

This enables the rotation function to retrieve the credentials from the master secret—then use the master secret credentials to change the credentials for the rotating secret.

Rotation fails because the secret value is not formatted as expected by the rotation function.

Rotation might also fail if you don't format the secret value as a JSON structure as expected by the rotation function. The rotation function you use determines the format used. For the details of what each rotation function requires for the secret value, see the Expected SecretString Value entry under the relevant rotation function at Secrets Manager rotation function templates.

For example, if you use the MySQL Single User rotation function, the SecretString text structure must look like this:

{ "engine": "mysql", "host": "<required: instance host name/resolvable DNS name>", "username": "<required: username>", "password": "<required: password>", "dbname": "<optional: database name. If not specified, defaults to None>", "port": "<optional: TCP port number. If not specified, defaults to 3306>" }

Secrets Manager says I successfully configured rotation, but the password isn't rotating

This can occur if there are network configuration issues that prevent the Lambda function from communicating with either your secured database/service or the Secrets Manager service endpoint, on the public Internet. If you run your database or service in a VPC, then you use one of two options for configuration:

To determine if this type of configuration issue caused the rotation failure, perform the following steps.

To diagnose connectivity issues between your rotation function and the database or Secrets Manager

  1. Open your logs by following the procedure I want to find the diagnostic logs for my Lambda rotation function.

  2. Examine the log files to look for indications that timeouts occurr between either the Lambda function and the AWS Secrets Manager service, or between the Lambda function and the secured database or service.

  3. For information about how to configure services and Lambda functions to interoperate within the VPC environment, see the Amazon Virtual Private Cloud documentation and the AWS Lambda Developer Guide .

Rotation fails with an "Internal failure" error message

When your rotation function generates a new password and attempts to store it in the database as a new set of credentials, you must ensure the password includes only characters valid for the specified database. The attempt to set the password for a user fails if the password includes characters that the database engine doesn't accept. This error appears as an "internal failure". Refer to the database documentation for a list of the characters you can use. Then, exclude all others by using the ExcludeCharacters parameter in the GetRandomPassword API call.

CloudTrail shows access-denied errors during rotation

When you configure rotation, if you let Secrets Manager create the rotation function for you, Secrets Manager automatically provides a policy attached to the function IAM role that grants the appropriate permissions. If you create a custom function, you need to grant the following permissions to the role attached to the function.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetRandomPassword", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "secretsmanager:UpdateSecretVersionStage", ], "Resource": "*" } ] }

Also, if your rotation uses separate master secret credentials to rotate this secret, then you must also grant permission to retrieve the secret value from the master secret. For more information, see My first rotation fails after I enable rotation. The combined policy might look like this:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAccessToSecretsManagerAPIs", "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetRandomPassword", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "secretsmanager:UpdateSecretVersionStage", ], "Resource": "*" }, { "Sid": "AllowAccessToMasterSecret", "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "MasterSecretArn" } ] }