Troubleshoot AWS Secrets Manager rotation

For many services, Secrets Manager uses a Lambda function to rotate secrets. For more information, see Rotation by Lambda function. The Lambda rotation function interacts with the database or service the secret is for as well as Secrets Manager. When rotation doesn't work the way you expect, you should first check the CloudWatch logs.

Note

Some services can manage secrets for you, including managing automatic rotation. For more information, see Managed rotation for AWS Secrets Manager secrets.

To view the CloudWatch logs for your Lambda function

1. Open the Secrets Manager console at https://console.aws.amazon.com/secretsmanager/.

2. Choose your secret, and then on the details page, under Rotation configuration, choose the Lambda rotation function. The Lambda console opens.

3. On the Monitor tab, choose Logs, and then choose View logs in CloudWatch.

   The CloudWatch console opens and displays the logs for your function.

No activity after "Found credentials in environment variables"

If there is no activity after "Found credentials in environment variables", and the task duration is long, for example the default Lambda timeout of 30000ms, then the Lambda function may be timing out while trying to reach the Secrets Manager endpoint.

Your Lambda rotation function must be able to access a Secrets Manager endpoint. If your Lambda function can access the internet, then you can use a public endpoint. To find an endpoint, see AWS Secrets Manager endpoints.

If your Lambda function runs in a VPC that doesn't have internet access, we recommend you configure Secrets Manager service private endpoints within your VPC. Your VPC can then intercept requests addressed to the public regional endpoint and redirect them to the private endpoint. For more information, see VPC endpoint.

Alternatively, you can enable your Lambda function to access a Secrets Manager public endpoint by adding a NAT gateway or an internet gateway to your VPC, which allows traffic from your VPC to reach the public endpoint. This exposes your VPC to more risk because an IP address for the gateway can be attacked from the public Internet.

No activity after "createSecret"

The following are issues that can cause rotation to stop after createSecret:

The VPC Network ACLs do not allow HTTPS traffic in and out.

For more information, see Control traffic to subnets using Network ACLs in the Amazon VPC User Guide.

Lambda function timeout configuration is too short to perform the task.

For more information, see Configuring Lambda function options in the AWS Lambda Developer Guide.

The Secrets Manager VPC endpoint does not allow the VPC CIDRs on ingress in the assigned security groups.

For more information, see Control traffic to resources using security groups in the Amazon VPC User Guide.

The Secrets Manager VPC endpoint policy does not allow Lambda to use the VPC endpoint.

For more information, see Using an AWS Secrets Manager VPC endpoint.

The secret uses alternating users rotation, the superuser secret is managed by Amazon RDS, and the Lambda function can't access the RDS API.

For alternating users rotation where the superuser secret is managed by another AWS service, the Lambda rotation function must be able to call the service endpoint to get the database connection information. We recommend that you configure a VPC endpoint for the database service. For more information, see:

• Amazon RDS API and interface VPC endpoints in the Amazon RDS User Guide.

• Working with VPC endpoints in the Amazon Redshift Management Guide.

Error: "Access to KMS is not allowed"

If you see ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed, the rotation function does not have permission to decrypt the secret using the KMS key that was used to encrypt the secret. There might be a condition in the permissions policy that limits the encryption context to a specific secret. For information about the required permission, see Policy statement for customer managed key.

Error: "Key is missing from secret JSON"

A Lambda rotation function requires the secret value to be in a specific JSON structure. If you see this error, then the JSON might be missing a key that the rotation function tried to access. For information about the JSON structure for each type of secret, see JSON structure of AWS Secrets Manager secrets .

Error: "setSecret: Unable to log into database"

The following are issues that can cause this error:

The rotation function can't access the database.

If the task duration is long, for example over 5000ms, then the Lambda rotation function might not be able to access the database over the network.

If your database or service is running on an Amazon EC2 instance in a VPC, we recommend that you configure your Lambda function to run in the same VPC. Then the rotation function can communicate directly with your service. For more information, see Configuring VPC access.

To allow the Lambda function to access the database or service, you must make sure that the security groups attached to your Lambda rotation function allow outbound connections to the database or service. You must also make sure that the security groups attached to your database or service allow inbound connections from the Lambda rotation function.

The credentials in the secret are incorrect.

If the task duration is short, then the Lambda rotation function might not be able to authenticate with the credentials in the secret. Check the credentials by logging in manually with the information in the AWSCURRENT and AWSPREVIOUS versions of the secret using the AWS CLI command get-secret-value.

The database uses scram-sha-256 to encrypt passwords.

If your database is Aurora PostgreSQL version 13 or later and uses scram-sha-256 to encrypt passwords, but the rotation function uses libpq version 9 or older which does not support scram-sha-256, then the rotation function can't connect to the database.

To determine which database users use scram-sha-256 encryption

• See Checking for users with non-SCRAM passwords in the blog SCRAM Authentication in RDS for PostgreSQL 13.

To determine which version of libpq your rotation function uses

1. On a Linux-based computer, on the Lambda console, navigate to your rotation function and download the deployment bundle. Uncompress the zip file into a work directory.

2. At a command line, in the work directory, run:

   readelf -a libpq.so.5 | grep RUNPATH

3. If you see the string PostgreSQL-9.4.x, or any major version less than 10, then the rotation function doesn't support scram-sha-256.

   • Output for a rotation function that doesn't support scram-sha-256:

     0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild-a1b2c/workspace/build/PostgreSQL/PostgreSQL-9.4.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild-a1b2c/workspace/src/PostgreSQL/build/private/install/lib]

   • Output for a rotation function that supports scram-sha-256:

     0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild-a1b2c/workspace/build/PostgreSQL/PostgreSQL-10.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild-a1b2c/workspace/src/PostgreSQL/build/private/install/lib]

Note

If you set up automatic secret rotation before December 30, 2021, your rotation function bundled an older version of libpq that doesn't support scram-sha-256. To support scram-sha-256, you need to recreate your rotation function.

The database requires SSL/TLS access.

If your database requires an SSL/TLS connection, but the rotation function uses an unencrypted connection, then the rotation function can't connect to the database. Rotation functions for Amazon RDS (except Oracle and Db2) and Amazon DocumentDB automatically use Secure Socket Layer (SSL) or Transport Layer Security (TLS) to connect to your database, if it is available. Otherwise they use an unencrypted connection.

Note

If you set up automatic secret rotation before December 20, 2021, your rotation function might be based on an older template that did not support SSL/TLS. To support connections that use SSL/TLS, you need to recreate your rotation function.

To determine when your rotation function was created

1. In the Secrets Manager console https://console.aws.amazon.com/secretsmanager/, open your secret. In the Rotation configuration section, under Lambda rotation function, you see the Lambda function ARN, for example, arn:aws:lambda:aws-region:123456789012:function:SecretsManagerMyRotationFunction . Copy the function name from the end of the ARN, in this example SecretsManagerMyRotationFunction .

2. In the AWS Lambda console https://console.aws.amazon.com/lambda/, under Functions, paste your Lambda function name in the search box, choose Enter, and then choose the Lambda function.

3. In the function details page, on the Configuration tab, under Tags, copy the value next to the key aws:cloudformation:stack-name.

4. In the AWS CloudFormation console https://console.aws.amazon.com/cloudformation, under Stacks, paste the key value in the search box, and then choose Enter.

5. The list of stacks filters so that only the stack that created the Lambda rotation function appears. In the Created date column, view the date the stack was created. This is the date the Lambda rotation function was created.

Error: "Unable to import module 'lambda_function'"

You might receive this error if you're running an earlier Lambda function that was automatically upgraded from Python 3.7 to a newer version of Python. To resolve the error, you can change the Lambda function version back to Python 3.7, and then Upgrade an existing rotation function from Python 3.7 to 3.9. For more information, see Why did my Secrets Manager Lambda function rotation fail with a “pg module not found“ error? in AWS re:Post.

Upgrade an existing rotation function from Python 3.7 to 3.9

Some rotation functions created before November 2022 used Python 3.7. The AWS SDK for Python stopped supporting Python 3.7 in December 2023. For more information, see Python support policy updates for AWS SDKs and Tools. To switch to a new rotation function that uses Python 3.9, you can add a runtime property to an existing rotation function or recreate the rotation function.

To find which Lambda rotation functions use Python 3.7

1. Sign in to the AWS Management Console and open the AWS Lambda console at https://console.aws.amazon.com/lambda/.

2. In the list of Functions, filter for SecretsManager.

3. In the filtered list of functions, under Runtime, look for Python 3.7.

To upgrade to Python 3.9:

• Option 1: Recreate the rotation function using AWS CloudFormation
• Option 2: Update the runtime for the existing rotation function using AWS CloudFormation
• Option 3: For AWS CDK users, upgrade the CDK library

Option 1: Recreate the rotation function using AWS CloudFormation

When you use the Secrets Manager console to turn on rotation, Secrets Manager uses AWS CloudFormation to create the necessary resources, including the Lambda rotation function. If you used the console to turn on rotation, or you created the rotation function using a AWS CloudFormation stack, you can use the same AWS CloudFormation stack to recreate the rotation function with a new name. The new function uses the more recent version of Python.

To find the AWS CloudFormation stack that created the rotation function

• On the Lambda function details page, on the Configuration tab, choose Tags. View the ARN next to aws:cloudformation:stack-id.

  The stack name is embedded in the ARN, as shown in the following example.

  • ARN: arn:aws:cloudformation:us-west-2:408736277230:stack/SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda-3CUDHZMDMBO8/79fc9050-2eef-11ed-80f0-021fb13c0537

  • Stack name: SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda

To recreate a rotation function (AWS CloudFormation)

1. In AWS CloudFormation, search for the stack by name, and then choose Update.

   If a dialog box appears recommending you update the root stack, choose Go to root stack, and then choose Update.

2. On the Update stack page, choose Edit template in designer, and then choose View in Designer.

3. In the designer, in the template code, in SecretRotationScheduleHostedRotationLambda, replace the value for "functionName": "SecretsManagerTestRotationRDS" with a new function name, for example in JSON, "functionName": "SecretsManagerTestRotationRDSupdated"

4. Continue through the AWS CloudFormation stack workflow and then choose Submit.

Option 2: Update the runtime for the existing rotation function using AWS CloudFormation

When you use the Secrets Manager console to turn on rotation, Secrets Manager uses AWS CloudFormation to create the necessary resources, including the Lambda rotation function. If you used the console to turn on rotation, or you created the rotation function using a AWS CloudFormation stack, you can use the same AWS CloudFormation stack to update the runtime for the rotation function.

To find the AWS CloudFormation stack that created the rotation function

• On the Lambda function details page, on the Configuration tab, choose Tags. View the ARN next to aws:cloudformation:stack-id.

  The stack name is embedded in the ARN, as shown in the following example.

  • ARN: arn:aws:cloudformation:us-west-2:408736277230:stack/SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda-3CUDHZMDMBO8/79fc9050-2eef-11ed-80f0-021fb13c0537

  • Stack name: SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda

To update the runtime for a rotation function (AWS CloudFormation)

1. In AWS CloudFormation, search for the stack by name, and then choose Update.

   If a dialog box appears recommending you update the root stack, choose Go to root stack, and then choose Update.

2. On the Update stack page, choose Edit template in designer, and then choose View in Designer.

3. In the designer, in the template JSON, for the SecretRotationScheduleHostedRotationLambda, under Properties, under Parameters, add "runtime": "python3.9"

4. Continue through the AWS CloudFormation stack workflow and then choose Submit.

Option 3: For AWS CDK users, upgrade the CDK library

If you used the AWS CDK prior to version v2.94.0 to set up rotation for your secret, you can update the Lambda function by upgrading to v2.94.0 or later. For more information, see the AWS Cloud Development Kit (AWS CDK) v2 Developer Guide.