Tutorial: Using AWS Lambda with Amazon DynamoDB streams
In this tutorial, you create a Lambda function to consume events from an Amazon DynamoDB stream.
Prerequisites
This tutorial assumes that you have some knowledge of basic Lambda operations and the Lambda console. If you haven't already, follow the instructions in Create a Lambda function with the console to create your first Lambda function.
To complete the following steps, you need the AWS Command Line Interface (AWS CLI) version 2. Commands and the expected output are listed in separate blocks:
aws --version
You should see the following output:
aws-cli/2.13.27 Python/3.11.6 Linux/4.14.328-248.540.amzn2.x86_64 exe/x86_64.amzn.2
For long commands, an escape character (\
) is used to split a command over multiple lines.
On Linux and macOS, use your preferred shell and package manager.
Note
In Windows, some Bash CLI commands that you commonly use with Lambda (such as zip
) are not supported by the operating system's built-in terminals.
To get a Windows-integrated version of Ubuntu and Bash, install the Windows Subsystem for Linux
Create the execution role
Create the execution role that gives your function permission to access AWS resources.
To create an execution role
-
Open the roles page
in the IAM console. -
Choose Create role.
-
Create a role with the following properties.
-
Trusted entity – Lambda.
-
Permissions – AWSLambdaDynamoDBExecutionRole.
-
Role name –
lambda-dynamodb-role
.
-
The AWSLambdaDynamoDBExecutionRole has the permissions that the function needs to read items from DynamoDB and write logs to CloudWatch Logs.
Create the function
Create a Lambda function that processes your DynamoDB events. The function code writes some of the incoming event data to CloudWatch Logs.
To create the function
-
Copy the sample code into a file named
example.js
. -
Create a deployment package.
zip function.zip example.js
-
Create a Lambda function with the
create-function
command.aws lambda create-function --function-name ProcessDynamoDBRecords \ --zip-file fileb://function.zip --handler example.handler --runtime nodejs18.x \ --role arn:aws:iam::
111122223333
:role/lambda-dynamodb-role
Test the Lambda function
In this step, you invoke your Lambda function manually using the invoke
AWS Lambda CLI command and
the following sample DynamoDB event. Copy the following into a file named input.txt
.
Example input.txt
{ "Records":[ { "eventID":"1", "eventName":"INSERT", "eventVersion":"1.0", "eventSource":"aws:dynamodb", "awsRegion":"us-east-1", "dynamodb":{ "Keys":{ "Id":{ "N":"101" } }, "NewImage":{ "Message":{ "S":"New item!" }, "Id":{ "N":"101" } }, "SequenceNumber":"111", "SizeBytes":26, "StreamViewType":"NEW_AND_OLD_IMAGES" }, "eventSourceARN":"stream-ARN" }, { "eventID":"2", "eventName":"MODIFY", "eventVersion":"1.0", "eventSource":"aws:dynamodb", "awsRegion":"us-east-1", "dynamodb":{ "Keys":{ "Id":{ "N":"101" } }, "NewImage":{ "Message":{ "S":"This item has changed" }, "Id":{ "N":"101" } }, "OldImage":{ "Message":{ "S":"New item!" }, "Id":{ "N":"101" } }, "SequenceNumber":"222", "SizeBytes":59, "StreamViewType":"NEW_AND_OLD_IMAGES" }, "eventSourceARN":"stream-ARN" }, { "eventID":"3", "eventName":"REMOVE", "eventVersion":"1.0", "eventSource":"aws:dynamodb", "awsRegion":"us-east-1", "dynamodb":{ "Keys":{ "Id":{ "N":"101" } }, "OldImage":{ "Message":{ "S":"This item has changed" }, "Id":{ "N":"101" } }, "SequenceNumber":"333", "SizeBytes":38, "StreamViewType":"NEW_AND_OLD_IMAGES" }, "eventSourceARN":"stream-ARN" } ] }
Run the following invoke
command.
aws lambda invoke --function-name ProcessDynamoDBRecords \ --cli-binary-format raw-in-base64-out \ --payload file://input.txt outputfile.txt
The cli-binary-format option is required if you're using AWS CLI version 2. To make this the default setting, run aws configure set cli-binary-format raw-in-base64-out
. For more information, see AWS CLI supported global command line options in the AWS Command Line Interface User Guide for Version 2.
The function returns the string message
in the response body.
Verify the output in the outputfile.txt
file.
Create a DynamoDB table with a stream enabled
Create an Amazon DynamoDB table with a stream enabled.
To create a DynamoDB table
-
Open the DynamoDB console
. -
Choose Create table.
-
Create a table with the following settings.
-
Table name –
lambda-dynamodb-stream
-
Primary key –
id
(string)
-
-
Choose Create.
To enable streams
-
Open the DynamoDB console
. -
Choose Tables.
-
Choose the lambda-dynamodb-stream table.
-
Under Exports and streams, choose DynamoDB stream details.
-
Choose Turn on.
-
For View type, choose Key attributes only.
-
Choose Turn on stream.
Write down the stream ARN. You need this in the next step when you associate the stream with your Lambda function. For more information on enabling streams, see Capturing table activity with DynamoDB Streams.
Add an event source in AWS Lambda
Create an event source mapping in AWS Lambda. This event source mapping associates the DynamoDB stream with your Lambda function. After you create this event source mapping, AWS Lambda starts polling the stream.
Run the following AWS CLI create-event-source-mapping
command. After the command runs, note
down the UUID. You'll need this UUID to refer to the event source mapping in any commands, for example, when
deleting the event source mapping.
aws lambda create-event-source-mapping --function-name ProcessDynamoDBRecords \ --batch-size 100 --starting-position LATEST --event-source
DynamoDB-stream-arn
This creates a mapping between the specified DynamoDB stream and the Lambda function. You can associate a DynamoDB stream with multiple Lambda functions, and associate the same Lambda function with multiple streams. However, the Lambda functions will share the read throughput for the stream they share.
You can get the list of event source mappings by running the following command.
aws lambda list-event-source-mappings
The list returns all of the event source mappings you created, and for each mapping it shows the
LastProcessingResult
, among other things. This field is used to provide an informative message if
there are any problems. Values such as No records processed
(indicates that AWS Lambda has not started
polling or that there are no records in the stream) and OK
(indicates AWS Lambda successfully read
records from the stream and invoked your Lambda function) indicate that there are no issues. If there are issues,
you receive an error message.
If you have a lot of event source mappings, use the function name parameter to narrow down the results.
aws lambda list-event-source-mappings --function-name ProcessDynamoDBRecords
Test the setup
Test the end-to-end experience. As you perform table updates, DynamoDB writes event records to the stream. As AWS Lambda polls the stream, it detects new records in the stream and invokes your Lambda function on your behalf by passing events to the function.
-
In the DynamoDB console, add, update, and delete items to the table. DynamoDB writes records of these actions to the stream.
-
AWS Lambda polls the stream and when it detects updates to the stream, it invokes your Lambda function by passing in the event data it finds in the stream.
-
Your function runs and creates logs in Amazon CloudWatch. You can verify the logs reported in the Amazon CloudWatch console.
Clean up your resources
You can now delete the resources that you created for this tutorial, unless you want to retain them. By deleting AWS resources that you're no longer using, you prevent unnecessary charges to your AWS account.
To delete the Lambda function
-
Open the Functions page
of the Lambda console. -
Select the function that you created.
-
Choose Actions, Delete.
-
Type
delete
in the text input field and choose Delete.
To delete the execution role
-
Open the Roles page
of the IAM console. -
Select the execution role that you created.
-
Choose Delete.
-
Enter the name of the role in the text input field and choose Delete.
To delete the DynamoDB table
-
Open the Tables page
of the DynamoDB console. -
Select the table you created.
-
Choose Delete.
-
Enter
delete
in the text box. -
Choose Delete table.