Tutorial: Using AWS Lambda with Amazon DocumentDB Streams

In this tutorial, you create a basic Lambda function that consumes events from an Amazon DocumentDB (with MongoDB compatibility) change stream. To complete this tutorial, you will go through the following stages:

• Set up your Amazon DocumentDB cluster, connect to it, and activate change streams on it.

• Create your Lambda function, and configure your Amazon DocumentDB cluster as an event source for your function.

• Test the end-to-end setup by inserting items into your Amazon DocumentDB database.

Prerequisites

Sign up for an AWS account

If you do not have an AWS account, complete the following steps to create one.

To sign up for an AWS account

1. Open https://portal.aws.amazon.com/billing/signup.

2. Follow the online instructions.

   Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.

   When you sign up for an AWS account, an AWS account root user is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform tasks that require root user access.

AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to https://aws.amazon.com/ and choosing My Account.

Create a user with administrative access

After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.

Secure your AWS account root user

1. Sign in to the AWS Management Console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.

   For help signing in by using root user, see Signing in as the root user in the AWS Sign-In User Guide.

2. Turn on multi-factor authentication (MFA) for your root user.

   For instructions, see Enable a virtual MFA device for your AWS account root user (console) in the IAM User Guide.

Create a user with administrative access

1. Enable IAM Identity Center.

   For instructions, see Enabling AWS IAM Identity Center in the AWS IAM Identity Center User Guide.

2. In IAM Identity Center, grant administrative access to a user.

   For a tutorial about using the IAM Identity Center directory as your identity source, see Configure user access with the default IAM Identity Center directory in the AWS IAM Identity Center User Guide.

Sign in as the user with administrative access

• To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.

  For help signing in using an IAM Identity Center user, see Signing in to the AWS access portal in the AWS Sign-In User Guide.

Assign access to additional users

1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.

   For instructions, see Create a permission set in the AWS IAM Identity Center User Guide.

2. Assign users to a group, and then assign single sign-on access to the group.

   For instructions, see Add groups in the AWS IAM Identity Center User Guide.

Install the AWS Command Line Interface

If you have not yet installed the AWS Command Line Interface, follow the steps at Installing or updating the latest version of the AWS CLI to install it.

The tutorial requires a command line terminal or shell to run commands. In Linux and macOS, use your preferred shell and package manager.

Note

In Windows, some Bash CLI commands that you commonly use with Lambda (such as zip) are not supported by the operating system's built-in terminals. To get a Windows-integrated version of Ubuntu and Bash, install the Windows Subsystem for Linux.

Create the AWS Cloud9 environment

Before creating the Lambda function, you need to create and configure your Amazon DocumentDB cluster. The steps to set up your cluster in this tutorial is based on the procedure in Get Started with Amazon DocumentDB.

Note

If you already have a Amazon DocumentDB cluster set up, ensure that you activate change streams and create the necessary interface VPC endpoints. Then, you can skip directly to the function creation steps.

First, create an AWS Cloud9 environment. You’ll use this environment throughout this tutorial to connect to and query your DocumentDB cluster.

To create an AWS Cloud9 environment

1. Open the Cloud9 console and choose Create environment.

2. Create an environment with the following configuration:

   • Under Details:

     • Name – DocumentDBCloud9Environment

     • Environment type – New EC2 instance

   • Under New EC2 instance:

     • Instance type – t2.micro (1 GiB RAM + 1 vCPU)

     • Platform – Amazon Linux 2

     • Timeout – 30 minutes

   • Under Network settings:

     • Connection – AWS Systems Manager (SSM)

     • Expand the VPC settings dropdown.

     • Amazon Virtual Private Cloud (VPC) – Choose your default VPC.

     • Subnet – No preference

   • Keep all other default settings.

3. Choose Create. Provisioning your new AWS Cloud9 environment can take several minutes.

Create the EC2 security group

Next, create a EC2 security group with rules that allow traffic between your DocumentDB cluster and your Cloud9 environment.

To create an EC2 security group

1. Open the EC2 console. Under Network and Security, choose Security groups.

2. Choose Create security group.

3. Create a security group with the following configuration:

   • Under Basic details:

     • Security group name – DocDBTutorial

     • Description – Security group for traffic between Cloud9 and DocumentDB.

     • VPC – Choose your default VPC.

   • Under Inbound rules, choose Add rule. Create a rule with the following configuration:

     • Type – Custom TCP

     • Port range – 27017

     • Source – Custom

     • In the search box next to Source, choose the security group for the AWS Cloud9 environment you created in the previous step. To see a list of available security groups, enter cloud9 in the search box. Choose the security group with the name aws-cloud9-<environment_name>.

   • Keep all other default settings.

4. Choose Create security group.

Create the DocumentDB cluster

In this step, you’ll create a DocumentDB cluster using the security group from the previous step.

To create a DocumentDB cluster

1. Open the DocumentDB console. Under Clusters, choose Create.

2. Create a cluster with the following configuration:

   • For Cluster type, choose Instance Based Cluster.

   • Under Configuration:

     • Engine version – 5.0.0

     • Instance class – db.t3.medium (free trial eligible)

     • Number of instances – 1.

   • Under Authentication:

     • Enter the Username and Password needed to connect to your cluster (same credentials as you used to create the secret in the previous step). In Confirm password, confirm your password.

   • Toggle on Show advanced settings.

   • Under Network settings:

     • Virtual Private Cloud (VPC) – Choose your default VPC.

     • Subnet group – default

     • VPC security groups – In addition to default (VPC), choose the DocDBTutorial (VPC) security group you created in the previous step.

   • Keep all other default settings.

3. Choose Create cluster. Provisioning your DocumentDB cluster can take several minutes.

Create the secret in Secrets Manager

To access your DocumentDB cluster manually, you must provide username and password credentials. For Lambda to access your cluster, you must provide a Secrets Manager secret that contains these same access credentials when setting up your event source mapping. In this step, you’ll create this secret.

To create the secret in Secrets Manager

1. Open the Secrets Manager console and choose Store a new secret.

2. For Choose secret type, choose the following options:

   • Under Basic details:

     • Secret type – Credentials for Amazon DocumentDB database

     • Under Credentials, enter the username and password you’ll use to access your DocumentDB cluster.

     • Database – Choose your DocumentDB cluster.

     • Choose Next.

3. For Configure secret, choose the following options:

   • Secret name – DocumentDBSecret

   • Choose Next.

4. Choose Next.

5. Choose Store.

6. Refresh the console to verify that you successfully stored the DocumentDBSecret secret.

Note down the Secret ARN of your secret. You’ll need it in a later step.

Install the mongo shell

In this step, you’ll install the mongo shell in your Cloud9 environment. The mongo shell is a command-line utility that you use to connect to and query your DocumentDB cluster.

To install the mongo shell on your Cloud9 environment

1. Open the Cloud9 console. Next to the DocumentDBCloud9Environment environment you created earlier, click on the Open link under the Cloud9 IDE column.

2. In the terminal window, create the MongoDB repository file with the following command:

   echo -e "[mongodb-org-5.0] \nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/5.0/x86_64/\ngpgcheck=1 \nenabled=1 \ngpgkey=https://www.mongodb.org/static/pgp/server-5.0.asc" | sudo tee /etc/yum.repos.d/mongodb-org-5.0.repo

3. Then, install the mongo shell with the following command:

   sudo yum install -y mongodb-org-shell

4. To encrypt data in transit, download the public key for Amazon DocumentDB. The following command downloads a file named global-bundle.pem:

   wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

Connect to the DocumentDB cluster

You’re now ready to connect to your DocumentDB cluster using the mongo shell.

To connect to your DocumentDB cluster

1. Open the DocumentDB console. Under Clusters, choose your cluster by choosing its cluster identifier.

2. In the Connectivity & security tab, under Connect to this cluster with the mongo shell, choose Copy.

3. In your Cloud9 environment, paste this command into the terminal. Replace <insertYourPassword> with the correct password.

After entering this command, if the command prompt becomes rs0:PRIMARY>, then you’re connected to your Amazon DocumentDB cluster.

Activate change streams

For this tutorial, you’ll track changes to the products collection of the docdbdemo database in your DocumentDB cluster. You do this by activating change streams. First, create the docdbdemo database and test it by inserting a record.

To create a new database within your cluster

1. In your Cloud9 environment, ensure that you’re still connected to your DocumentDB cluster.

2. In the terminal window, use the following command to create a new database called docdbdemo:

   use docdbdemo

3. Then, use the following command to insert a record into docdbdemo:

   db.products.insert({"hello":"world"})

   You should see output that looks like this:

   WriteResult({ "nInserted" : 1 })

4. Use the following command to list all databases:

   show dbs

   Ensure that your output contains the docdbdemo database:

   docdbdemo  0.000GB

Next, activate change streams on the products collection of the docdbdemo database using the following command:

db.adminCommand({modifyChangeStreams: 1,
    database: "docdbdemo",
    collection: "products", 
    enable: true});

You should see output that looks like this:

{ "ok" : 1, "operationTime" : Timestamp(1680126165, 1) }

Create interface VPC endpoints

Next, create interface VPC endpoints to ensure that Lambda and Secrets Manager (used later to store our cluster access credentials) can connect to your default VPC.

To create interface VPC endpoints

1. Open the VPC console. In the left menu, under Virtual private cloud, choose Endpoints.

2. Choose Create endpoint. Create an endpoint with the following configuration:

   • For Name tag, enter lambda-default-vpc.

   • For Service category, choose AWS services.

   • For Services, enter lambda in the search box. Choose the service with format com.amazonaws.<region>.lambda.

   • For VPC, choose your default VPC.

   • For Subnets, check the boxes next to each availability zone. Choose the correct subnet ID for each availability zone.

   • For IP address type, select IPv4.

   • For Security groups, choose the default VPC security group (Group name of default), and the security group you created earlier (Group name of DocDBTutorial).

   • Keep all other default settings.

   • Choose Create endpoint.

3. Again, choose Create endpoint. Create an endpoint with the following configuration:

   • For Name tag, enter secretsmanager-default-vpc.

   • For Service category, choose AWS services.

   • For Services, enter secretsmanager in the search box. Choose the service with format com.amazonaws.<region>.secretsmanager.

   • For VPC, choose your default VPC.

   • For Subnets, check the boxes next to each availability zone. Choose the correct subnet ID for each availability zone.

   • For IP address type, select IPv4.

   • For Security groups, choose the default VPC security group (Group name of default), and the security group you created earlier (Group name of DocDBTutorial).

   • Keep all other default settings.

   • Choose Create endpoint.

This completes the cluster setup portion of this tutorial.

Create the execution role

In the next set of steps, you’ll create your Lambda function. First, you need to create the execution role that gives your function permission to access your cluster. You do this by creating an IAM policy first, then attaching this policy to an IAM role.

To create IAM policy

1. Open the Policies page in the IAM console and choose Create policy.

2. Choose the JSON tab. In the following policy, replace the Secrets Manager resource ARN in the final line of the statement with your secret ARN from earlier, and copy the policy into the editor.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "LambdaESMNetworkingAccess",
               "Effect": "Allow",
               "Action": [
                   "ec2:CreateNetworkInterface",
                   "ec2:DescribeNetworkInterfaces",
                   "ec2:DescribeVpcs",
                   "ec2:DeleteNetworkInterface",
                   "ec2:DescribeSubnets",
                   "ec2:DescribeSecurityGroups",
                   "kms:Decrypt"
               ],
               "Resource": "*"
           },
           {
               "Sid": "LambdaDocDBESMAccess",
               "Effect": "Allow",
               "Action": [
                   "rds:DescribeDBClusters",
                   "rds:DescribeDBClusterParameters",
                   "rds:DescribeDBSubnetGroups"
               ],
               "Resource": "*"
           },
           {
               "Sid": "LambdaDocDBESMGetSecretValueAccess",
               "Effect": "Allow",
               "Action": [
                   "secretsmanager:GetSecretValue"
               ],
               "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:DocumentDBSecret"
           }
       ]
   }

3. Choose Next: Tags, then choose Next: Review.

4. For Name, enter AWSDocumentDBLambdaPolicy.

5. Choose Create policy.

To create the IAM role

1. Open the Roles page in the IAM console and choose Create role.

2. For Select trusted entity, choose the following options:

   • Trusted entity type – AWS service

   • Use case – Lambda

   • Choose Next.

3. For Add permissions, choose the AWSDocumentDBLambdaPolicy policy you just created, as well as the AWSLambdaBasicExecutionRole to give your function permissions to write to Amazon CloudWatch Logs.

4. Choose Next.

5. For Role name, enter AWSDocumentDBLambdaExecutionRole.

6. Choose Create role.

Create the Lambda function

The following example code receives a DocumentDB event input and processes the message that it contains.

Go

SDK for Go V2

Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using Go.

// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

package main

import (
	"context"
	"encoding/json"
	"fmt"

	"github.com/aws/aws-lambda-go/lambda"
)

type Event struct {
	Events []Record `json:"events"`
}

type Record struct {
	Event struct {
		OperationType string `json:"operationType"`
		NS            struct {
			DB   string `json:"db"`
			Coll string `json:"coll"`
		} `json:"ns"`
		FullDocument interface{} `json:"fullDocument"`
	} `json:"event"`
}

func main() {
	lambda.Start(handler)
}

func handler(ctx context.Context, event Event) (string, error) {
	fmt.Println("Loading function")
	for _, record := range event.Events {
		logDocumentDBEvent(record)
	}

	return "OK", nil
}

func logDocumentDBEvent(record Record) {
	fmt.Printf("Operation type: %s\n", record.Event.OperationType)
	fmt.Printf("db: %s\n", record.Event.NS.DB)
	fmt.Printf("collection: %s\n", record.Event.NS.Coll)
	docBytes, _ := json.MarshalIndent(record.Event.FullDocument, "", "  ")
	fmt.Printf("Full document: %s\n", string(docBytes))
}

JavaScript

SDK for JavaScript (v3)

Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using JavaScript.

console.log('Loading function');
exports.handler = async (event, context) => {
    event.events.forEach(record => {
        logDocumentDBEvent(record);
    });
    return 'OK';
};

const logDocumentDBEvent = (record) => {
    console.log('Operation type: ' + record.event.operationType);
    console.log('db: ' + record.event.ns.db);
    console.log('collection: ' + record.event.ns.coll);
    console.log('Full document:', JSON.stringify(record.event.fullDocument, null, 2));
};

Python

SDK for Python (Boto3)

Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using Python.

import json

def lambda_handler(event, context):
    for record in event.get('events', []):
        log_document_db_event(record)
    return 'OK'

def log_document_db_event(record):
    event_data = record.get('event', {})
    operation_type = event_data.get('operationType', 'Unknown')
    db = event_data.get('ns', {}).get('db', 'Unknown')
    collection = event_data.get('ns', {}).get('coll', 'Unknown')
    full_document = event_data.get('fullDocument', {})

    print(f"Operation type: {operation_type}")
    print(f"db: {db}")
    print(f"collection: {collection}")
    print("Full document:", json.dumps(full_document, indent=2))

Ruby

SDK for Ruby

Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using Ruby.

require 'json'

def lambda_handler(event:, context:)
  event['events'].each do |record|
    log_document_db_event(record)
  end
  'OK'
end

def log_document_db_event(record)
  event_data = record['event'] || {}
  operation_type = event_data['operationType'] || 'Unknown'
  db = event_data.dig('ns', 'db') || 'Unknown'
  collection = event_data.dig('ns', 'coll') || 'Unknown'
  full_document = event_data['fullDocument'] || {}

  puts "Operation type: #{operation_type}"
  puts "db: #{db}"
  puts "collection: #{collection}"
  puts "Full document: #{JSON.pretty_generate(full_document)}"
end

To create the Lambda function

1. Copy the sample code into a file named index.js.

2. Create a deployment package with the following command.

   zip function.zip index.js

3. Use the following CLI command to create the function. Replace us-east-1 with the region, and 123456789012 with your account ID.

   aws lambda create-function --function-name ProcessDocumentDBRecords \
       --zip-file fileb://function.zip --handler index.handler --runtime nodejs20.x \
       --region us-east-1 \
       --role arn:aws:iam::123456789012:role/AWSDocumentDBLambdaExecutionRole

Create the Lambda event source mapping

Create the event source mapping that associates your DocumentDB change stream with your Lambda function. After you create this event source mapping, AWS Lambda immediately starts polling the stream.

To create the event source mapping

1. Open the Functions page in the Lambda console.

2. Choose the ProcessDocumentDBRecords function you created earlier.

3. Choose the Configuration tab, then choose Triggers in the left menu.

4. Choose Add trigger.

5. Under Trigger configuration, for the source, select DocumentDB.

6. Create the event source mapping with the following configuration:

   • DocumentDB cluster – Choose the cluster you created earlier.

   • Database name – docdbdemo

   • Collection name – products

   • Batch size – 1

   • Starting position – Latest

   • Authentication – BASIC_AUTH

   • Secrets Manager key – Choose the DocumentDBSecret you just created.

   • Batch window – 1

   • Full document configuration – UpdateLookup

7. Choose Add. Creating your event source mapping can take a few minutes.

Test your function - manual invoke

To test that you created your function and event source mapping correctly, invoke your function using the invoke command. To do this, first copy the following event JSON into a file called input.txt:

{
  "eventSourceArn": "arn:aws:rds:us-east-1:123456789012:cluster:canaryclusterb2a659a2-qo5tcmqkcl03",
  "events": [
    {
      "event": {
        "_id": {
          "_data": "0163eeb6e7000000090100000009000041e1"
        },
        "clusterTime": {
          "$timestamp": {
            "t": 1676588775,
            "i": 9
          }
        },
        "documentKey": {
          "_id": {
            "$oid": "63eeb6e7d418cd98afb1c1d7"
          }
        },
        "fullDocument": {
          "_id": {
            "$oid": "63eeb6e7d418cd98afb1c1d7"
          },
          "anyField": "sampleValue"
        },
        "ns": {
          "db": "docdbdemo",
          "coll": "products"
        },
        "operationType": "insert"
      }
    }
  ],
  "eventSource": "aws:docdb"
}

Then, use the following command to invoke your function with this event:

aws lambda invoke --function-name ProcessDocumentDBRecords \
    --cli-binary-format raw-in-base64-out \
    --region us-east-1 \
    --payload file://input.txt out.txt

You should see a response that looks like the following:

{
   "StatusCode": 200,
   "ExecutedVersion": "$LATEST"
}

You can verify that your function successfully processed the event by checking CloudWatch Logs.

To verify manual invocation via CloudWatch Logs

1. Open the Functions page in the Lambda console.

2. Choose the Monitor tab, then choose View CloudWatch logs. This takes you to the specific log group associated with your function in the CloudWatch console.

3. Choose the most recent log stream. Within the log messages, you should see the event JSON.

Test your function - insert a record

Test your end-to-end setup by interacting directly with your DocumentDB database. In the next set of steps, you’ll insert a record, update it, then delete it.

To insert a record

1. Reconnect to your DocumentDB cluster in your Cloud9 environment.

2. Use this command to ensure that you’re currently using the docdbdemo database:

   use docdbdemo

3. Insert a record into the products collection of the docdbdemo database:

   db.products.insert({"name":"Pencil", "price": 1.00})

Test your function - update a record

Next, update the record you just inserted with the following command:

db.products.update(
    { "name": "Pencil" },
    { $set: { "price": 0.50 }}
)

Verify that your function successfully processed this event by checking CloudWatch Logs.

Test your function - delete a record

Finally, delete the record you just updated with the following command:

db.products.remove( { "name": "Pencil" } )

Verify that your function successfully processed this event by checking CloudWatch Logs.

Clean up your resources

You can now delete the resources that you created for this tutorial, unless you want to retain them. By deleting AWS resources that you're no longer using, you prevent unnecessary charges to your AWS account.

To delete the Lambda function

1. Open the Functions page of the Lambda console.

2. Select the function that you created.

3. Choose Actions, Delete.

4. Type delete in the text input field and choose Delete.

To delete the execution role

1. Open the Roles page of the IAM console.

2. Select the execution role that you created.

3. Choose Delete.

4. Enter the name of the role in the text input field and choose Delete.

To delete the VPC endpoints

1. Open the VPC console. In the left menu, under Virtual private cloud, choose Endpoints.

2. Select the endpoints you created.

3. Choose Actions, Delete VPC endpoints.

4. Enter delete in the text input field.

5. Choose Delete.

To delete the Amazon DocumentDB cluster

1. Open the DocumentDB console.

2. Choose the DocumentDB cluster you created for this tutorial, and disable deletion protection.

3. In the main Clusters page, choose your DocumentDB cluster again.

4. Choose Actions, Delete.

5. For Create final cluster snapshot, select No.

6. Enter delete in the text input field.

7. Choose Delete.

To delete the secret in Secrets Manager

1. Open the Secrets Manager console.

2. Choose the secret you created for this tutorial.

3. Choose Actions, Delete secret.

4. Choose Schedule deletion.

To delete the Amazon EC2 security group

1. Open the EC2 console. Under Network and Security, choose Security groups.

2. Select the security group you created for this tutorial.

3. Choose Actions, Delete security groups.

4. Choose Delete.

To delete the Cloud9 environment

1. Open the Cloud9 console.

2. Select the environment you created for this tutorial.

3. Choose Delete.

4. Enter delete in the text input field.

5. Choose Delete.