Cache secrets using AWS Lambda extensions - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsRelated resourcesAdditional informationAttachments

Cache secrets using AWS Lambda extensions

Created by HARI OHM PRASATH RAJAGOPAL (AWS), Karan Shah (AWS), and Giridhar Shyam Sankararaman (AWS)

Environment: PoC or pilot

Technologies: Modernization; Cloud-native; Serverless; Software development & testing

Workload: Open-source

AWS services: AWS Lambda; AWS Secrets Manager

Summary

This pattern demonstrates how to host an HTTP server that caches secrets required for an AWS Lambda function. The Lambda extension does the following:

  • Hosts a local HTTP server, which is invoked by a Lambda function to retrieve the secret instead of retrieving it directly from AWS Secrets Manager.

  • Uses a config.yaml file inside the Lambda function to get the list of secrets that must be cached in memory.

  • Refreshes the secrets cache based on the value (in minutes) set in the environment variable CACHE_TIMEOUT. If no value is specified, it defaults to 10.

This extension is helpful in the following use cases:

  • If the Lambda function needs access to Secrets Manager for reading a secret, the secret can get cached as part of the extension startup instead of being fetched as part of Lambda's first request. This helps to avoid a cold start. 

  • The Lambda function specifies which secrets are cached by using the configuration file (config.yaml), which is part of the deployment package.

  • The Lambda function allows the expiration of in-memory cache, which can be configured as a Lambda environment variable.

  • The business can maintain the cache layer in-memory instead of using a /tmp storage for reading and writing values in Secrets Manager.

Prerequisites and limitations

Prerequisites 

  • An active AWS account.

  • AWS Command Line Interface (AWS CLI) version 1.7 or later, installed and configured on macOS, Linux, or Windows.

  • Zip utility installed in the local system.

  • The code that is in the Attachments section, which contains the code for creating a Lambda extension as a layer. Details are in the Additional information section.

Architecture

Architecture

The following diagram provides a high-level view of all the components.

  1. Secrets are cached based on the config.yaml file defined inside the deployment package of the Lambda function, and the cache is refreshed based on environment variables.

  2. Based on the keys specified inside config.yaml, the secrets are read and cached in-memory from Secrets Manager.

  3. The Lambda function reads the secret by making an HTTP GET call to the extension, and the extension returns the value from the in-memory cache.

Initializing the extension and reading secrets from the cache

The following sequence diagram shows the initialization of the Lambda extension and how the Lambda function reads cached secrets by using the HTTP server that is hosted inside the extension.

The sequence of events inside the Lambda function and the extension includes the following.

  1. The extension is initialized first before the Lambda function, based on the lifecycle. As part of the initialization, the extension reads the config.yaml file from the Lambda function under the default path (/var/task) and gets the list of secrets that need to be cached.

  2. When the extension has a list of keys that needs to be fetched, it invokes Secrets Manager to get all the secrets (specified in config.yaml) and stores them in memory.

  3. The extension starts a local HTTP server.

  4. To get the secret value, the Lambda function makes an HTTP GET:/cache/<secretname> call to the extension.

    Note: The web server is made available locally to the Lambda runtime environment. The communication between the Lambda function and the web server is in HTTP. It is assumed that this is fine, as this example is to demonstrate the capability to communicate to the cache locally.

  5. The extension returns the cached value to the Lambda function based on the secretname path parameter.

Technology

  • AWS Lambda

  • AWS Secrets Manager

  • HTTP web server

Tools

Tools

  • AWS CLI – AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services.

  • AWS Lambda – AWS Lambda supports running code without provisioning or managing servers.

  • AWS Secrets Manager – AWS Secrets Manager enables the replacement of hardcoded credentials, including passwords, with an API call to Secrets Manager to retrieve the secret. Because Secrets Manager can automatically rotate the secret according to a schedule, you can replace long-term secrets with short-term ones, reducing the risk of compromise.

  • Git client – Git provides GUI tools, or you can use the command line or a desktop tool to check out the required artifacts from GitHub.

  • Node.js – Node.js is an event-driven JavaScript runtime environment designed for building scalable network applications. The runtime for Node.js should be installed on the local system.

Epics

TaskDescriptionSkills required
Create a secret in Secrets Manager.

Run the command that is in the Additional information section. The command assumes that you placed your secret, such as this example JavaScript Object Notation (JSON) text structure {"username":"janedoe","password":" aBC1D2*!3EE"}, in a file named mycreds.json.

Developer, AWS systems administrator
TaskDescriptionSkills required
Create a new Lambda function.

On the AWS Management Console, choose Lambda and then choose Create function. Create a new Node.js runtime Lambda function with the name Secrets-Extension-Lambda-Test. Increase the memory of Lambda to 1200 MB and Timeout to 30 seconds.

Developer, AWS systems administrator
Assign a policy to the Lambda IAM role.

Choose the Configuration tab and navigate to the Permission tab. Under Execution role, choose the Role name link to open the attached role in AWS Identity and Access Management (IAM). Attach the SecretsManagerReadWrite policy to the role. 

Note: As a best practice, create an IAM policy with read-only permission on the secret that needs to be cached, and assign it to the IAM role.

Developer, AWS systems administrator
Update the handler code.

Choose the Code tab and update the function with the code included in the Additional information section. The code invokes the local server hosted inside the Lambda extension to read the value of the secret instead of directly going to Secrets Manager.

Developer, AWS systems administrator
Create a secrets cache configuration file.

In the Lambda function Code tab, create a new file named config.yaml under the root directory (alongside index.js) with the contents specified in the Additional information section. Ensure that the formatting of the yaml file is syntactically appropriate (as provided in the Additional information section). If you are copying the file from the attached secrets.zip file, ensure that the contents are formatted appropriately before you deploy the Lambda function.

The value secret_now is the name of the secret that you created in the previous step and that will be cached by the extension. If you want more secrets to be cached, you can keep adding them here. Choose the Deploy button.

Developer, AWS systems administrator
Create the cache, rebuild timeout, and update the Lambda runtime.

Create a new environment variable CACHE_TIMEOUT, and set the value in minutes. The cache will be refreshed based on this value. 

Note: If the environment value is not found, the cache gets refreshed every 10 minutes.

Developer, AWS systems administrator
TaskDescriptionSkills required
Deploy the Lambda extension.

To deploy the extension and associate the layer to the Lambda function, run the command that is in the Additional information section.

Developer, AWS systems administrator
TaskDescriptionSkills required
Invoke the Lambda function.

To invoke the Lambda function, use the command that is in the Additional information section. The function should return StatusCode:200.

Developer, AWS systems administrator
TaskDescriptionSkills required
Verify Lambda extension invocation.

On the Amazon CloudWatch console, navigate to Logs, Log Groups. Select the log group /aws/lambda/Secrets-Extension-Lambda-Test. View the log stream to see the runtime log with Response from cache followed by the user name and password stored in mycred.json.

Developer, AWS systems administrator

Related resources

Additional information

Create secret

The following command assumes that you placed your secret, such as this example JSON text structure {"username":"janedoe","password":" aBC1D2*!3EE"}, in a file named mycreds.json.

aws secretsmanager create-secret --name secret_now --secret-string file://mycreds.json

Update the handler code

An example with the following code is in example-function/index.js in the zipped files that are provided in the Attachments section.

exports.handler = function(event, context, callback) {
    const https = require('http')
    const options = {
        hostname: 'localhost',
        port: 8080,
        path: '/cache/secret_now',
        method: 'GET'
    }
    const req = https.request(options, res => {
        res.on('data', d => {
            console.log("Response from cache: "+d);
            return d;
        })
    })
    req.on('error', error => {
        console.error(error)
    })
    req.end()
};

Create a secrets cache configuration file

Create a config.yaml file in the Lambda function Code tab under the root directory. In the example, secret_now is the name of the secret that was created in the previous step and will be cached by the extension. To cache additional secrets, add them to this file.

SecretManagers:
  - secrets:
    - secret_now

Deploy the Lambda extension

> chmod +x deploy.sh
> ./deploy.sh

Invoke the Lambda function

aws lambda invoke \
 --function-name "Secrets-Extension-Lambda-Test" \
 --payload '{"payload": "hello"}' /tmp/invoke-result \
 --cli-binary-format raw-in-base64-out \
 --log-type Tail

The function should return "StatusCode": 200.

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip