Tutorial: Using Lambda with Amazon SQS
In this tutorial, you create a Lambda function that consumes messages from an Amazon Simple Queue Service (Amazon SQS) queue. The Lambda function runs whenever a new message is added to the queue. The function writes the messages to an Amazon CloudWatch Logs stream. The following diagram shows the AWS resources you use to complete the tutorial.
To complete this tutorial, you carry out the following steps:
-
Create a Lambda function that writes messages to CloudWatch Logs.
-
Create an Amazon SQS queue.
-
Create a Lambda event source mapping. The event source mapping reads the Amazon SQS queue and invokes your Lambda function when a new message is added.
-
Test the setup by adding messages to your queue and monitoring the results in CloudWatch Logs.
Prerequisites
If you do not have an AWS account, complete the following steps to create one.
To sign up for an AWS account
Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.
When you sign up for an AWS account, an AWS account root user is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform tasks that require root user access.
AWS sends you a confirmation email after the sign-up process is
complete. At any time, you can view your current account activity and manage your account by
going to https://aws.amazon.com/
After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.
Secure your AWS account root user
-
Sign in to the AWS Management Console
as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password. For help signing in by using root user, see Signing in as the root user in the AWS Sign-In User Guide.
-
Turn on multi-factor authentication (MFA) for your root user.
For instructions, see Enable a virtual MFA device for your AWS account root user (console) in the IAM User Guide.
Create a user with administrative access
-
Enable IAM Identity Center.
For instructions, see Enabling AWS IAM Identity Center in the AWS IAM Identity Center User Guide.
-
In IAM Identity Center, grant administrative access to a user.
For a tutorial about using the IAM Identity Center directory as your identity source, see Configure user access with the default IAM Identity Center directory in the AWS IAM Identity Center User Guide.
Sign in as the user with administrative access
-
To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.
For help signing in using an IAM Identity Center user, see Signing in to the AWS access portal in the AWS Sign-In User Guide.
Assign access to additional users
-
In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.
For instructions, see Create a permission set in the AWS IAM Identity Center User Guide.
-
Assign users to a group, and then assign single sign-on access to the group.
For instructions, see Add groups in the AWS IAM Identity Center User Guide.
If you have not yet installed the AWS Command Line Interface, follow the steps at Installing or updating the latest version of the AWS CLI to install it.
The tutorial requires a command line terminal or shell to run commands. In Linux and macOS, use your preferred shell and package manager.
Note
In Windows, some Bash CLI commands that you commonly use with Lambda (such as zip
) are not supported by the operating system's built-in terminals.
To get a Windows-integrated version of Ubuntu and Bash, install the Windows Subsystem for Linux
Create the execution role
An execution role is an AWS Identity and Access Management (IAM) role that grants a Lambda function permission to access AWS services and resources. To allow your function to read items from Amazon SQS, attach the AWSLambdaSQSQueueExecutionRole permissions policy.
To create an execution role and attach an Amazon SQS permissions policy
-
Open the Roles page
of the IAM console. -
Choose Create role.
-
For Trusted entity type, choose AWS service.
-
For Use case, choose Lambda.
-
Choose Next.
-
In the Permissions policies search box, enter
AWSLambdaSQSQueueExecutionRole
. -
Select the AWSLambdaSQSQueueExecutionRole policy, and then choose Next.
-
Under Role details, for Role name, enter
lambda-sqs-role
, then choose Create role.
After role creation, note down the Amazon Resource Name (ARN) of your execution role. You'll need it in later steps.
Create the function
Create a Lambda function that processes your Amazon SQS messages. The function code logs the body of the Amazon SQS message to CloudWatch Logs.
This tutorial uses the Node.js 18.x runtime, but we've also provided example code in other runtime languages. You can select the tab in the following box to see code for the runtime you're interested in. The JavaScript code you'll use in this step is in the first example shown in the JavaScript tab.
To create a Node.js Lambda function
-
Create a directory for the project, and then switch to that directory.
mkdir sqs-tutorial cd sqs-tutorial
-
Copy the sample JavaScript code into a new file named
index.js
. -
Create a deployment package using the following
zip
command.zip function.zip index.js
-
Create a Lambda function using the create-function
AWS CLI command. For the role
parameter, enter the ARN of the execution role that you created earlier.Note
The Lambda function and the Amazon SQS queue must be in the same AWS Region.
aws lambda create-function --function-name ProcessSQSRecord \ --zip-file fileb://function.zip --handler index.handler --runtime nodejs18.x \
--role arn:aws:iam::
111122223333
:role/lambda-sqs-role
Test the function
Invoke your Lambda function manually using the invoke
AWS CLI command and a sample Amazon SQS
event.
To invoke the Lambda function with a sample event
-
Save the following JSON as a file named
input.json
. This JSON simulates an event that Amazon SQS might send to your Lambda function, where"body"
contains the actual message from the queue. In this example, the message is"test"
.Example Amazon SQS event
This is a test event—you don't need to change the message or the account number.
{ "Records": [ { "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d", "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...", "body": "test", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1545082649183", "SenderId": "AIDAIENQZJOLO23YVJ4VO", "ApproximateFirstReceiveTimestamp": "1545082649185" }, "messageAttributes": {}, "md5OfBody": "098f6bcd4621d373cade4e832627b4f6", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-1:111122223333:my-queue", "awsRegion": "us-east-1" } ] }
-
Run the following invoke
AWS CLI command. This command returns CloudWatch logs in the response. For more information about retrieving logs, see Access logs with the AWS CLI. aws lambda invoke --function-name ProcessSQSRecord --payload file://input.json out --log-type Tail \ --query 'LogResult' --output text --cli-binary-format raw-in-base64-out | base64 --decode
The cli-binary-format option is required if you're using AWS CLI version 2. To make this the default setting, run
aws configure set cli-binary-format raw-in-base64-out
. For more information, see AWS CLI supported global command line options in the AWS Command Line Interface User Guide for Version 2. -
Find the
INFO
log in the response. This is where the Lambda function logs the message body. You should see logs that look like this:2023-09-11T22:45:04.271Z 348529ce-2211-4222-9099-59d07d837b60 INFO Processed message test 2023-09-11T22:45:04.288Z 348529ce-2211-4222-9099-59d07d837b60 INFO done
Create an Amazon SQS queue
Create an Amazon SQS queue that the Lambda function can use as an event source. The Lambda function and the Amazon SQS queue must be in the same AWS Region.
To create a queue
-
Open the Amazon SQS console
. -
Choose Create queue.
-
Enter a name for the queue. Leave all other options at the default settings.
-
Choose Create queue.
After creating the queue, note down its ARN. You need this in the next step when you associate the queue with your Lambda function.
Configure the event source
Connect the Amazon SQS queue to your Lambda function by creating an event source mapping. The event source mapping reads the Amazon SQS queue and invokes your Lambda function when a new message is added.
To create a mapping between your Amazon SQS queue and your Lambda function, use the create-event-source-mapping
aws lambda create-event-source-mapping --function-name ProcessSQSRecord --batch-size 10 \ --event-source-arn arn:aws:sqs:
us-east-1:111122223333:my-queue
To get a list of your event source mappings, use the list-event-source-mappings
aws lambda list-event-source-mappings --function-name ProcessSQSRecord
Send a test message
To send an Amazon SQS message to the Lambda function
-
Open the Amazon SQS console
. -
Choose the queue that you created earlier.
-
Choose Send and receive messages.
-
Under Message body, enter a test message, such as "this is a test message."
-
Choose Send message.
Lambda polls the queue for updates. When there is a new message, Lambda invokes your function with this new event data from the queue. If the function handler returns without exceptions, Lambda considers the message successfully processed and begins reading new messages in the queue. After successfully processing a message, Lambda automatically deletes it from the queue. If the handler throws an exception, Lambda considers the batch of messages not successfully processed, and Lambda invokes the function with the same batch of messages.
Check the CloudWatch logs
To confirm that the function processed the message
Open the Functions page
of the Lambda console. -
Choose the ProcessSQSRecord function.
-
Choose Monitor.
-
Choose View CloudWatch logs.
-
In the CloudWatch console, choose the Log stream for the function.
-
Find the
INFO
log. This is where the Lambda function logs the message body. You should see the message that you sent from the Amazon SQS queue. Example:2023-09-11T22:49:12.730Z b0c41e9c-0556-5a8b-af83-43e59efeec71 INFO
Processed message this is a test message.
Clean up your resources
You can now delete the resources that you created for this tutorial, unless you want to retain them. By deleting AWS resources that you're no longer using, you prevent unnecessary charges to your AWS account.
To delete the execution role
-
Open the Roles page
of the IAM console. -
Select the execution role that you created.
-
Choose Delete.
-
Enter the name of the role in the text input field and choose Delete.
To delete the Lambda function
-
Open the Functions page
of the Lambda console. -
Select the function that you created.
-
Choose Actions, Delete.
-
Type
delete
in the text input field and choose Delete.
To delete the Amazon SQS queue
-
Sign in to the AWS Management Console and open the Amazon SQS console at https://console.aws.amazon.com/sqs/
. -
Select the queue you created.
-
Choose Delete.
-
Enter
confirm
in the text input field. -
Choose Delete.