Namespace Amazon.CDK.AWS.Lambda.EventSources
AWS Lambda Event Sources
An event source mapping is an AWS Lambda resource that reads from an event source and invokes a Lambda function. You can use event source mappings to process items from a stream or queue in services that don't invoke Lambda functions directly. Lambda provides event source mappings for the following services. Read more about lambda event sources here.
This module includes classes that allow using various AWS services as event
sources for AWS Lambda via the high-level lambda.addEventSource(source)
API.
NOTE: In most cases, it is also possible to use the resource APIs to invoke an AWS Lambda function. This library provides a uniform API for all Lambda event sources regardless of the underlying mechanism they use.
The following code sets up a lambda function with an SQS queue event source -
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;
var queue = new Queue(this, "MyQueue");
var eventSource = new SqsEventSource(queue);
fn.AddEventSource(eventSource);
var eventSourceId = eventSource.EventSourceMappingId;
var eventSourceMappingArn = eventSource.EventSourceMappingArn;
The eventSourceId
property contains the event source id. This will be a
token that will resolve to the final value at the time of
deployment.
The eventSourceMappingArn
property contains the event source mapping ARN. This will be a
token that will resolve to the final value at the time of
deployment.
SQS
Amazon Simple Queue Service (Amazon SQS) allows you to build asynchronous workflows. For more information about Amazon SQS, see Amazon Simple Queue Service. You can configure AWS Lambda to poll for these messages as they arrive and then pass the event to a Lambda function invocation. To view a sample event, see Amazon SQS Event.
To set up Amazon Simple Queue Service as an event source for AWS Lambda, you first create or update an Amazon SQS queue and select custom values for the queue parameters. The following parameters will impact Amazon SQS's polling behavior:
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;
var queue = new Queue(this, "MyQueue", new QueueProps {
VisibilityTimeout = Duration.Seconds(30)
});
fn.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps {
BatchSize = 10, // default
MaxBatchingWindow = Duration.Minutes(5),
ReportBatchItemFailures = true
}));
S3
You can write Lambda functions to process S3 bucket events, such as the object-created or object-deleted events. For example, when a user uploads a photo to a bucket, you might want Amazon S3 to invoke your Lambda function so that it reads the image and creates a thumbnail for the photo.
You can use the bucket notification configuration feature in Amazon S3 to configure the event source mapping, identifying the bucket events that you want Amazon S3 to publish and which Lambda function to invoke.
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;
var bucket = new Bucket(this, "mybucket");
fn.AddEventSource(new S3EventSource(bucket, new S3EventSourceProps {
Events = new [] { EventType.OBJECT_CREATED, EventType.OBJECT_REMOVED },
Filters = new [] { new NotificationKeyFilter { Prefix = "subdir/" } }
}));
In the example above, S3EventSource
is accepting Bucket
type as parameter.
However, Functions like from_bucket_name
and from_bucket_arn
will return IBucket
and is not compliant with S3EventSource
. If this is the case, please consider using
S3EventSourceV2
instead, this class accepts IBucket
.
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;
var bucket = Bucket.FromBucketName(this, "Bucket", "bucket-name");
fn.AddEventSource(new S3EventSourceV2(bucket, new S3EventSourceProps {
Events = new [] { EventType.OBJECT_CREATED, EventType.OBJECT_REMOVED },
Filters = new [] { new NotificationKeyFilter { Prefix = "subdir/" } }
}));
SNS
You can write Lambda functions to process Amazon Simple Notification Service notifications. When a message is published to an Amazon SNS topic, the service can invoke your Lambda function by passing the message payload as a parameter. Your Lambda function code can then process the event, for example publish the message to other Amazon SNS topics, or send the message to other AWS services.
This also enables you to trigger a Lambda function in response to Amazon CloudWatch alarms and other AWS services that use Amazon SNS.
For an example event, see Appendix: Message and JSON Formats and Amazon SNS Sample Event. For an example use case, see Using AWS Lambda with Amazon SNS from Different Accounts.
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.Lambda.EventSources;
Topic topic;
Function fn;
var deadLetterQueue = new Queue(this, "deadLetterQueue");
fn.AddEventSource(new SnsEventSource(topic, new SnsEventSourceProps {
FilterPolicy = new Dictionary<string, object> { },
DeadLetterQueue = deadLetterQueue
}));
When a user calls the SNS Publish API on a topic that your Lambda function is subscribed to, Amazon SNS will call Lambda to invoke your function asynchronously. Lambda will then return a delivery status. If there was an error calling Lambda, Amazon SNS will retry invoking the Lambda function up to three times. After three tries, if Amazon SNS still could not successfully invoke the Lambda function, then Amazon SNS will send a delivery status failure message to CloudWatch.
DynamoDB Streams
You can write Lambda functions to process change events from a DynamoDB Table. An event is emitted to a DynamoDB stream (if configured) whenever a write (Put, Delete, Update) operation is performed against the table. See Using AWS Lambda with Amazon DynamoDB for more information about configuring Lambda function event sources with DynamoDB.
To process events with a Lambda function, first create or update a DynamoDB table and enable a stream
specification. Then, create a DynamoEventSource
and add it to your Lambda function. The following parameters will impact Amazon DynamoDB's polling behavior:
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda.EventSources;
Table table;
Function fn;
var deadLetterQueue = new Queue(this, "deadLetterQueue");
fn.AddEventSource(new DynamoEventSource(table, new DynamoEventSourceProps {
StartingPosition = StartingPosition.TRIM_HORIZON,
BatchSize = 5,
BisectBatchOnError = true,
OnFailure = new SqsDlq(deadLetterQueue),
RetryAttempts = 10
}));
Kinesis
You can write Lambda functions to process streaming data in Amazon Kinesis Streams. For more information about Amazon Kinesis, see Amazon Kinesis Service. To learn more about configuring Lambda function event sources with kinesis and view a sample event, see Amazon Kinesis Event.
To set up Amazon Kinesis as an event source for AWS Lambda, you first create or update an Amazon Kinesis stream and select custom values for the event source parameters. The following parameters will impact Amazon Kinesis's polling behavior:
using Amazon.CDK.AWS.Kinesis;
using Amazon.CDK.AWS.Lambda.EventSources;
Function myFunction;
var stream = new Stream(this, "MyStream");
myFunction.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps {
BatchSize = 100, // default
StartingPosition = StartingPosition.TRIM_HORIZON
}));
Kafka
You can write Lambda functions to process data either from Amazon MSK or a self managed Kafka cluster.
The following code sets up Amazon MSK as an event source for a lambda function. Credentials will need to be configured to access the MSK cluster, as described in Username/Password authentication.
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.Lambda.EventSources;
Function myFunction;
// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";
// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
// The secret that allows access to your MSK cluster
// You still have to make sure that it is associated with your cluster as described in the documentation
var secret = new Secret(this, "Secret", new SecretProps { SecretName = "AmazonMSK_KafkaSecret" });
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
ClusterArn = clusterArn,
Topic = topic,
Secret = secret,
BatchSize = 100, // default
StartingPosition = StartingPosition.TRIM_HORIZON
}));
The following code sets up a self managed Kafka cluster as an event source. Username and password based authentication will need to be set up as described in Managing access and permissions.
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.Lambda.EventSources;
// The secret that allows access to your self hosted Kafka cluster
Secret secret;
Function myFunction;
// The list of Kafka brokers
var bootstrapServers = new [] { "kafka-broker:9092" };
// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
// (Optional) The consumer group id to use when connecting to the Kafka broker. If omitted the UUID of the event source mapping will be used.
var consumerGroupId = "my-consumer-group-id";
myFunction.AddEventSource(new SelfManagedKafkaEventSource(new SelfManagedKafkaEventSourceProps {
BootstrapServers = bootstrapServers,
Topic = topic,
ConsumerGroupId = consumerGroupId,
Secret = secret,
BatchSize = 100, // default
StartingPosition = StartingPosition.TRIM_HORIZON
}));
If your self managed Kafka cluster is only reachable via VPC also configure vpc
vpcSubnets
and securityGroup
.
You can specify event filtering
for managed and self managed Kafka clusters using the filters
property:
using Amazon.CDK.AWS.Lambda.EventSources;
Function myFunction;
// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";
// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
ClusterArn = clusterArn,
Topic = topic,
StartingPosition = StartingPosition.TRIM_HORIZON,
Filters = new [] { FilterCriteria.Filter(new Dictionary<string, object> {
{ "stringEquals", FilterRule.IsEqual("test") }
}) }
}));
You can also specify an S3 bucket as an "on failure" destination:
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.S3;
IBucket bucket;
Function myFunction;
// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";
// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
var s3OnFailureDestination = new S3OnFailureDestination(bucket);
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
ClusterArn = clusterArn,
Topic = topic,
StartingPosition = StartingPosition.TRIM_HORIZON,
OnFailure = s3OnFailureDestination
}));
Roadmap
Eventually, this module will support all the event sources described under Supported Event Sources in the AWS Lambda Developer Guide.
Classes
ApiEventSource | |
AuthenticationMethod | The authentication method to use with SelfManagedKafkaEventSource. |
BaseStreamEventSourceProps | The set of properties for streaming event sources shared by Dynamo, Kinesis and Kafka. |
DynamoEventSource | Use an Amazon DynamoDB stream as an event source for AWS Lambda. |
DynamoEventSourceProps | |
KafkaEventSourceProps | Properties for a Kafka event source. |
KinesisEventSource | Use an Amazon Kinesis stream as an event source for AWS Lambda. |
KinesisEventSourceProps | |
ManagedKafkaEventSource | Use a MSK cluster as a streaming source for AWS Lambda. |
ManagedKafkaEventSourceProps | Properties for a MSK event source. |
S3EventSource | Use S3 bucket notifications as an event source for AWS Lambda. |
S3EventSourceProps | |
S3EventSourceV2 | S3EventSourceV2 Use S3 bucket notifications as an event source for AWS Lambda. |
S3OnFailureDestination | An S3 dead letter bucket destination configuration for a Lambda event source. |
SelfManagedKafkaEventSource | Use a self hosted Kafka installation as a streaming source for AWS Lambda. |
SelfManagedKafkaEventSourceProps | Properties for a self managed Kafka cluster event source. |
SnsDlq | An SNS dead letter queue destination configuration for a Lambda event source. |
SnsEventSource | Use an Amazon SNS topic as an event source for AWS Lambda. |
SnsEventSourceProps | Properties forwarded to the Lambda Subscription. |
SqsDlq | An SQS dead letter queue destination configuration for a Lambda event source. |
SqsEventSource | Use an Amazon SQS queue as an event source for AWS Lambda. |
SqsEventSourceProps | |
StreamEventSource | Use an stream as an event source for AWS Lambda. |
StreamEventSourceProps | The set of properties for streaming event sources shared by Dynamo and Kinesis. |
Interfaces
IBaseStreamEventSourceProps | The set of properties for streaming event sources shared by Dynamo, Kinesis and Kafka. |
IDynamoEventSourceProps | |
IKafkaEventSourceProps | Properties for a Kafka event source. |
IKinesisEventSourceProps | |
IManagedKafkaEventSourceProps | Properties for a MSK event source. |
IS3EventSourceProps | |
ISelfManagedKafkaEventSourceProps | Properties for a self managed Kafka cluster event source. |
ISnsEventSourceProps | Properties forwarded to the Lambda Subscription. |
ISqsEventSourceProps | |
IStreamEventSourceProps | The set of properties for streaming event sources shared by Dynamo and Kinesis. |