Using Lambda with Amazon MSK - AWS Lambda

Using Lambda with Amazon MSK

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters. It supports multiple open-source versions of Kafka.

When you create an Amazon MSK cluster, you receive the required hosting and connection information of the cluster. This information includes the Kafka cluster hostname, topic name, SASL/SCRAM user name and password, and bootstrap server host-port pairs.

To support your Kafka cluster on Amazon MSK, you might need to create Amazon Virtual Private Cloud (Amazon VPC) networking components. For more information, see Using Amazon MSK as an event source for AWS Lambda on the AWS Compute Blog.

Managing access and permissions for an Amazon MSK cluster

Lambda polls your Apache Kafka topic partitions for new records and invokes your Lambda function synchronously. To update other AWS resources that your cluster uses, your Lambda function—as well as your AWS Identity and Access Management (IAM) users and roles—must have permission to perform these actions.

This page describes how to grant permission to Lambda and other users of your Amazon MSK cluster.

Required Lambda function permissions

To read records from your Amazon MSK cluster on your behalf, your Lambda function's execution role must have permission. You can either add the AWS managed policy AWSLambdaMSKExecutionRole to your execution role, or create a custom policy with permission to perform the following actions:

Adding a policy to your execution role

Follow these steps to add the AWS managed policy AWSLambdaMSKExecutionRole to your execution role using the IAM console.

To add an AWS managed policy

  1. Open the Policies page of the IAM console.

  2. In the search box, enter the policy name (AWSLambdaMSKExecutionRole).

  3. Select the policy from the list, and then choose Policy actions, Attach.

  4. Select your execution role from the list, and then choose Attach policy.

Granting users access with an IAM policy

By default, IAM users and roles don't have permission to perform Amazon MSK API operations. To grant access to users in your organization or account, you might need an identity-based policy. For more information, see Amazon Managed Streaming for Apache Kafka Identity-Based Policy Examples in the Amazon Managed Streaming for Apache Kafka Developer Guide.

Using SASL/SCRAM authentication

Amazon MSK supports Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM) authentication. You can control access to your Amazon MSK clusters by setting up user name and password authentication using an AWS Secrets Manager secret. For more information, see Using Username and Password Authentication with AWS Secrets Manager in the Amazon Managed Streaming for Apache Kafka Developer Guide.

Adding an Amazon MSK cluster as an event source

You can use a Lambda function to process records from your Apache Kafka cluster when the cluster is configured as an event source. To create an event source mapping, you can add your Kafka cluster as a Lambda function trigger using the Lambda console, AWS SDK, or AWS Command Line Interface (AWS CLI).

This section describes how to add your Kafka cluster and topic as a function trigger using the Lambda console or AWS CLI.

Prerequisites

VPC configuration

To get Apache Kafka records from Amazon MSK brokers, Lambda must have access to the Amazon Virtual Private Cloud (Amazon VPC) resources associated with your MSK cluster. To meet Amazon VPC access requirements, do one of the following:

Your Amazon VPC security groups must be configured with the following rules (at minimum):

  • Inbound rules – Allow all traffic on all ports for the security group specified as your event source.

  • Outbound rules – Allow all traffic on all ports for all destinations.

Note

Your Amazon VPC configuration is discoverable through the Amazon MSK API, and doesn't need to be configured in your create-event-source-mapping setup.

Adding an Amazon MSK cluster using the Lambda console

Follow these steps to add your Amazon MSK cluster and a Kafka topic as a trigger for your Lambda function.

To add an MSK trigger to your Lambda function (console)

  1. Open the Functions page of the Lambda console.

  2. Choose the name of your Lambda function.

  3. Under Function overview, choose Add trigger.

  4. Under Trigger configuration, choose the MSK trigger type.

  5. Configure the remaining options, and then choose Add.

Adding an Amazon MSK cluster using the AWS CLI

Use the following example AWS CLI commands to create and view an Amazon MSK trigger for your Lambda function.

Creating a trigger using the AWS CLI

The following example uses the create-event-source-mapping AWS CLI command to map a Lambda function named my-kafka-function to a Kafka topic named AWSKafkaTopic. The topic's starting position is set to latest.

aws lambda create-event-source-mapping --event-source-arn arn:aws:kafka:us-west-2:arn:aws:kafka:us-west-2:111111111111:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 --topics AWSKafkaTopic --starting-position LATEST --function-name my-kafka-function

For more information, see the CreateEventSourceMapping API reference documentation.

Viewing the status using the AWS CLI

The following example uses the get-event-source-mapping AWS CLI command to describe the status of the event source mapping that you created.

aws lambda get-event-source-mapping --uuid 6d9bce8e-836b-442c-8070-74e77903c815