DynamoDB Streams and Time to Live
You can back up, or otherwise process, items that are deleted by Time to Live (TTL) by enabling Amazon DynamoDB Streams on the table and processing the streams records of the expired items. For more information, see Reading and processing a stream.
The streams record contains a user identity field
Records[
.<index>
].userIdentity
Items that are deleted by the Time to Live process after expiration have the following fields:
-
Records[
<index>
].userIdentity.type"Service"
-
Records[
<index>
].userIdentity.principalId"dynamodb.amazonaws.com"
Note
When you use TTL in a global table, the region the TTL was performed in will have the userIdentity
field set. This field won't be set in other regions when the delete is replicated.
The following JSON shows the relevant portion of a single streams record.
"Records": [ { ... "userIdentity": { "type": "Service", "principalId": "dynamodb.amazonaws.com" } ... } ]
Using DynamoDB Streams and Lambda to archive TTL deleted items
Combining DynamoDB Time to Live (TTL),
DynamoDB Streams, and AWS LambdaGetRecords
API calls on your DynamoDB stream when using Lambda to
consume events, and Lambda can provide event filtering by identifying JSON patterns in a
stream event. With event-pattern content filtering, you can define up to five different
filters to control which events are sent to Lambda for processing. This helps reduce
invocations of your Lambda functions, simplifies code, and reduces overall cost.
While DynamoDB Streams contains all data modifications, such as Create
,
Modify
, and Remove
actions, this can result in unwanted
invocations of your archive Lambda function. For example, say you have a table with 2
million data modifications per hour flowing into the stream, but less than 5 percent of
these are item deletes that will expire through the TTL process and need to be archived.
With Lambda event source filters, the Lambda function will only invoke 100,000
times per hour. The result with event filtering is that you’re charged only for the
needed invocations instead of the 2 million invocations you would have without event
filtering.
Event filtering is applied to the Lambda event source mapping, which is a resource that reads from a chosen event—the DynamoDB stream—and invokes a Lambda function. In the following diagram, you can see how a Time to Live deleted item is consumed by a Lambda function using streams and event filters.
DynamoDB Time to Live event filter pattern
Adding the following JSON to your event source mapping filter criteria allows invocation of your Lambda function only for TTL deleted items:
{ "Filters": [ { "Pattern": { "userIdentity": { "type": ["Service"], "principalId": ["dynamodb.amazonaws.com"] } } } ] }
Create an AWS Lambda event source mapping
Use the following code snippets to create a filtered event source mapping which you can connect to a table's DynamoDB stream. Each code block includes the event filter pattern.