Menu
Amazon DynamoDB
Developer Guide (API Version 2012-08-10)

Tutorial: Processing New Items in a DynamoDB Table

In this tutorial, you will create an AWS Lambda trigger to process a stream from a DynamoDB table.

The scenario for this tutorial is Woofer, a simple social network. Woofer users communicate using barks (short text messages) that are sent to other Woofer users. The following diagram shows the components and workflow for this application:

  1. A user writes an item to a DynamoDB table (BarkTable). Each item in the table represents a bark.

  2. A new stream record is written to reflect that a new item has been added to BarkTable.

  3. The new stream record triggers an AWS Lambda function (publishNewBark).

  4. If the stream record indicates that a new item was added to BarkTable, the Lambda function reads the data from the stream record and publishes a message to a topic in Amazon Simple Notification Service (Amazon SNS).

  5. The message is received by subscribers to the Amazon SNS topic. (In this tutorial, the only subscriber is an email address.)

Before You Begin

This tutorial uses the AWS Command Line Interface. If you have not done so already, follow the instructions in the AWS Command Line Interface User Guide install and configure the AWS CLI.

Step 1: Create a DynamoDB Table With a Stream Enabled

In this step, you will create a DynamoDB table (BarkTable) to store all of the barks from Woofer users. The primary key is composed of Username (partition key) and Timestamp (sort key). Both of these attributes are of type string.

BarkTable will have a stream enabled. Later in this tutorial, you will create a trigger by associating an AWS Lambda function with the stream.

  1. Type the following command to create the table:

    Copy
    aws dynamodb create-table \ --table-name BarkTable \ --attribute-definitions AttributeName=Username,AttributeType=S AttributeName=Timestamp,AttributeType=S \ --key-schema AttributeName=Username,KeyType=HASH AttributeName=Timestamp,KeyType=RANGE \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \ --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES
  2. In the output, look for the LatestStreamArn:

    Copy
    ... "LatestStreamArn": "arn:aws:dynamodb:region:accountID:table/BarkTable/stream/timestamp ...

    Make a note of the region and the accountID, because you will need them for the other steps in this tutorial.

Step 2: Create a Lambda Execution Role

In this step, you will create an IAM role (WooferLambdaRole) and assign permissions to it. This role will be used by the Lambda function that you will create in Step 4: Create and Test a Lambda Function.

You will also create a policy for the role. The policy will contain all of the permissions that the Lambda function will need at runtime.

  1. Create a file named trust-relationship.json with the following contents:

    Copy
    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
  2. Type the following command to create WooferLambdaRole:

    Copy
    aws iam create-role --role-name WooferLambdaRole \ --path "/service-role/" \ --assume-role-policy-document file://trust-relationship.json
  3. Create a file named role-policy.json with the following contents. (Replace region and accountID with your AWS region and account ID.)

    Copy
    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "lambda:InvokeFunction", "Resource": "arn:aws:lambda:region:accountID:function:publishNewBark*" }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:region:accountID:*" }, { "Effect": "Allow", "Action": [ "dynamodb:DescribeStream", "dynamodb:GetRecords", "dynamodb:GetShardIterator", "dynamodb:ListStreams" ], "Resource": "arn:aws:dynamodb:region:accountID:table/BarkTable/stream/*" }, { "Effect": "Allow", "Action": [ "sns:Publish" ], "Resource": [ "*" ] } ] }

    The policy has four statements, to allow WooferLambdaRole to do the following:

    • Execute a Lambda function (publishNewBark). You will create the function later in this tutorial.

    • Access CloudWatch Logs. The Lambda function will write diagnostics to CloudWatch Logs at runtime.

    • Read data from the DynamoDB stream for BarkTable.

    • Publish messages to Amazon SNS.

  4. Type the following command to attach the policy to WooferLambdaRole:

    Copy
    aws iam put-role-policy --role-name WooferLambdaRole \ --policy-name WooferLambdaRolePolicy \ --policy-document file://role-policy.json

Step 3: Create an Amazon SNS Topic

In this step, you will create an Amazon SNS topic (wooferTopic) and subscribe an email address to it. Your Lambda function will use this topic to publish new barks from Woofer users.

  1. Type the following command to create a new Amazon SNS topic:

    Copy
    aws sns create-topic --name wooferTopic
  2. Type the following command to subscribe an email address to wooferTopic. (Replace region and accountID with your AWS region and account ID, and replace example@example.com with a valid email address.)

    Copy
    aws sns subscribe \ --topic-arn arn:aws:sns:region:accountID:wooferTopic \ --protocol email \ --notification-endpoint example@example.com
  3. Amazon SNS will send a confirmation message to your email address. Click the Confirm subscription link in that message to complete the subscription process.

Step 4: Create and Test a Lambda Function

In this step, you will create an AWS Lambda function (publishNewBark) to process stream records from BarkTable.

The publishNewBark function processes only the stream events that correspond to new items in BarkTable. The function reads data from such an event, and then invokes Amazon SNS to publish it.

  1. Create a file named publishNewBark.js with the following contents:. (Replace region and accountID with your AWS region and account ID.)

    Copy
    'use strict'; var AWS = require("aws-sdk"); var sns = new AWS.SNS(); exports.handler = (event, context, callback) => { event.Records.forEach((record) => { console.log('Stream record: ', JSON.stringify(record, null, 2)); if (record.eventName == 'INSERT') { var who = JSON.stringify(record.dynamodb.NewImage.Username.S); var when = JSON.stringify(record.dynamodb.NewImage.Timestamp.S); var what = JSON.stringify(record.dynamodb.NewImage.Message.S); var params = { Subject: 'A new bark from ' + who, Message: 'Woofer user ' + who + ' barked the following at ' + when + ':\n\n ' + what, TopicArn: 'arn:aws:sns:region:accountID:wooferTopic' }; sns.publish(params, function(err, data) { if (err) { console.error("Unable to send message. Error JSON:", JSON.stringify(err, null, 2)); } else { console.log("Results from sending message: ", JSON.stringify(data, null, 2)); } }); } }); callback(null, `Successfully processed ${event.Records.length} records.`); };
  2. Create a zip file to contain publishNewBark.js. If you have the zip command-line utility you can type the following command to do this:

    Copy
    zip publishNewBark.zip publishNewBark.js
  3. When you create the Lambda function, you specify the ARN for WooferLambdaRole, which you created in Step 2: Create a Lambda Execution Role. Type the following command to retrieve this ARN:

    Copy
    aws iam get-role --role-name WooferLambdaRole

    In the output, look for the ARN for WooferLambdaRole:

    Copy
    ... "Arn": "arn:aws:iam::region:role/service-role/WooferLambdaRole" ...

    Now type the following command to create the Lambda function. (Replace roleARN with the ARN for WooferLambdaRole.)

    Copy
    aws lambda create-function \ --region us-east-1 \ --function-name publishNewBark \ --zip-file fileb://publishNewBark.zip \ --role roleARN \ --handler publishNewBark.handler \ --timeout 5 \ --runtime nodejs4.3
  4. Now you will test publishNewBark to verify that it works. To do this, you will provide input that resembles a real record from DynamoDB Streams.

    Create a file named payload.json with the following contents.

    Copy
    { "Records": [ { "eventID": "7de3041dd709b024af6f29e4fa13d34c", "eventName": "INSERT", "eventVersion": "1.1", "eventSource": "aws:dynamodb", "awsRegion": "us-west-2", "dynamodb": { "ApproximateCreationDateTime": 1479499740, "Keys": { "Timestamp": { "S": "2016-11-18:12:09:36" }, "Username": { "S": "John Doe" } }, "NewImage": { "Timestamp": { "S": "2016-11-18:12:09:36" }, "Message": { "S": "This is a bark from the Woofer social network" }, "Username": { "S": "John Doe" } }, "SequenceNumber": "13021600000000001596893679", "SizeBytes": 112, "StreamViewType": "NEW_IMAGE" }, "eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/BarkTable/stream/2016-11-16T20:42:48.104" } ] }

    Type the following command to test the publishNewBark function:

    Copy
    aws lambda invoke --function-name publishNewBark --payload file://payload.json output.txt

    If the test was successful, you will see the following output:

    Copy
    { "StatusCode": 200 }

    In addition, the output.txt file will contain this text:

    Copy
    "Successfully processed 1 records."

    You will also receive a new email message within a few minutes.

    Note

    AWS Lambda writes diagnostic information to Amazon CloudWatch Logs. If you encounter errors with your Lambda function, you can use these diagnostics for troubleshooting purposes:

    1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

    2. In the navigation pane, choose Logs.

    3. Choose the following log group: /aws/lambda/publishNewBark

    4. Choose the latest log stream to view the output (and errors) from the function.

Step 5: Create and Test a Trigger

In Step 4: Create and Test a Lambda Function, you tested the Lambda function to ensure that it ran correctly. In this step, you will create a trigger by associating the Lambda function (publishNewBark) with an event source (the BarkTable stream).

  1. When you create the trigger, you will need to specify the ARN for the BarkTable stream. Type the following command to retrieve this ARN:

    Copy
    aws dynamodb describe-table --table-name BarkTable

    In the output, look for the LatestStreamArn:

    Copy
    ... "LatestStreamArn": "arn:aws:dynamodb:region:accountID:table/BarkTable/stream/timestamp ...
  2. Type the following command to create the trigger. (Replace streamARN with the l stream ARN.)

    Copy
    aws lambda create-event-source-mapping \ --region us-east-1 \ --function-name publishNewBark \ --event-source streamARN \ --batch-size 1 \ --starting-position TRIM_HORIZON
  3. You will now test the trigger. Type the following command to add an item to BarkTable:

    Copy
    aws dynamodb put-item \ --table-name BarkTable \ --item Username={S="Jane Doe"},Timestamp={S="2016-11-18:14:32:17"},Message={S="Testing...1...2...3"}

    You should receive a new email message within a few minutes.

  4. Go to the DynamoDB console and add a few more items to BarkTable. You must specify values for the Username and Timestamp attributes. (You should also specify a value for Message, even though it is not required.) You should receive a new email message for each item you add to BarkTable.

    The Lambda function processes only new items that you add to BarkTable. If you update or delete an item in the table, the function does nothing.

Note

AWS Lambda writes diagnostic information to Amazon CloudWatch Logs. If you encounter errors with your Lambda function, you can use these diagnostics for troubleshooting purposes.

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. In the navigation pane, choose Logs.

  3. Choose the following log group: /aws/lambda/publishNewBark

  4. Choose the latest log stream to view the output (and errors) from the function.