Tutorial: Using Lambda with Amazon SQS - AWS Lambda

Tutorial: Using Lambda with Amazon SQS

In this tutorial, you create a Lambda function that consumes messages from an Amazon Simple Queue Service (Amazon SQS) queue. The Lambda function runs whenever a new message is added to the queue. The function writes the messages to an Amazon CloudWatch Logs stream. The following diagram shows the AWS resources you use to complete the tutorial.


        Diagram showing Amazon SQS message, Lambda function, and CloudWatch Logs stream

To complete this tutorial, you carry out the following steps:

  1. Create a Lambda function that writes messages to CloudWatch Logs.

  2. Create an Amazon SQS queue.

  3. Create a Lambda event source mapping. The event source mapping reads the Amazon SQS queue and invokes your Lambda function when a new message is added.

  4. Test the setup by adding messages to your queue and monitoring the results in CloudWatch Logs.

Prerequisites

If you do not have an AWS account, complete the following steps to create one.

To sign up for an AWS account
  1. Open https://portal.aws.amazon.com/billing/signup.

  2. Follow the online instructions.

    Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.

    When you sign up for an AWS account, an AWS account root user is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to an administrative user, and use only the root user to perform tasks that require root user access.

AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to https://aws.amazon.com/ and choosing My Account.

After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.

Secure your AWS account root user
  1. Sign in to the AWS Management Console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.

    For help signing in by using root user, see Signing in as the root user in the AWS Sign-In User Guide.

  2. Turn on multi-factor authentication (MFA) for your root user.

    For instructions, see Enable a virtual MFA device for your AWS account root user (console) in the IAM User Guide.

Create an administrative user
  1. Enable IAM Identity Center.

    For instructions, see Enabling AWS IAM Identity Center in the AWS IAM Identity Center User Guide.

  2. In IAM Identity Center, grant administrative access to an administrative user.

    For a tutorial about using the IAM Identity Center directory as your identity source, see Configure user access with the default IAM Identity Center directory in the AWS IAM Identity Center User Guide.

Sign in as the administrative user
  • To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.

    For help signing in using an IAM Identity Center user, see Signing in to the AWS access portal in the AWS Sign-In User Guide.

If you have not yet installed the AWS Command Line Interface, follow the steps at Installing or updating the latest version of the AWS CLI to install it.

The tutorial requires a command line terminal or shell to run commands. In Linux and macOS, use your preferred shell and package manager.

Note

In Windows, some Bash CLI commands that you commonly use with Lambda (such as zip) are not supported by the operating system's built-in terminals. To get a Windows-integrated version of Ubuntu and Bash, install the Windows Subsystem for Linux.

Create the execution role


        Step 1 create the execution role

An execution role is an AWS Identity and Access Management (IAM) role that grants a Lambda function permission to access AWS services and resources. To allow your function to read items from Amazon SQS, attach the AWSLambdaSQSQueueExecutionRole permissions policy.

To create an execution role and attach an Amazon SQS permissions policy
  1. Open the Roles page of the IAM console.

  2. Choose Create role.

  3. For Trusted entity type, choose AWS service.

  4. For Use case, choose Lambda.

  5. Choose Next.

  6. In the Permissions policies search box, enter AWSLambdaSQSQueueExecutionRole.

  7. Select the AWSLambdaSQSQueueExecutionRole policy, and then choose Next.

  8. Under Role details, for Role name, enter lambda-sqs-role, then choose Create role.

After role creation, note down the Amazon Resource Name (ARN) of your execution role. You'll need it in later steps.

Create the function


        Step 2 create the Lambda function

Create a Lambda function that processes your Amazon SQS messages. The function code logs the body of the Amazon SQS message to CloudWatch Logs.

This tutorial uses the Node.js 18.x runtime, but we've also provided example code in other runtime languages. You can select the tab in the following box to see code for the runtime you're interested in. The JavaScript code you'll use in this step is in the first example shown in the JavaScript tab.

.NET
AWS SDK for .NET
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using .NET.

using Amazon.Lambda.Core; using Amazon.Lambda.SQSEvents; // Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class. [assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))] namespace SqsIntegrationSampleCode { public async Task FunctionHandler(SQSEvent evnt, ILambdaContext context) { foreach (var message in evnt.Records) { await ProcessMessageAsync(message, context); } context.Logger.LogInformation("done"); } private async Task ProcessMessageAsync(SQSEvent.SQSMessage message, ILambdaContext context) { try { context.Logger.LogInformation($"Processed message {message.Body}"); // TODO: Do interesting work based on the new message await Task.CompletedTask; } catch (Exception e) { //You can use Dead Letter Queue to handle failures. By configuring a Lambda DLQ. context.Logger.LogError($"An error occurred"); throw; } } }
Go
SDK for Go V2
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using Go.

package integration_sqs_to_lambda import ( "fmt" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) func handler(event events.SQSEvent) error { for _, record := range event.Records { err := processMessage(record) if err != nil { return err } } fmt.Println("done") return nil } func processMessage(record events.SQSMessage) error { fmt.Printf("Processed message %s\n", record.Body) // TODO: Do interesting work based on the new message return nil } func main() { lambda.Start(handler) }
Java
SDK for Java 2.x
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using Java.

import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.RequestHandler; import com.amazonaws.services.lambda.runtime.events.SQSEvent; import com.amazonaws.services.lambda.runtime.events.SQSEvent.SQSMessage; public class Function implements RequestHandler<SQSEvent, Void> { @Override public Void handleRequest(SQSEvent sqsEvent, Context context) { for (SQSMessage msg : sqsEvent.getRecords()) { processMessage(msg, context); } context.getLogger().log("done"); return null; } private void processMessage(SQSMessage msg, Context context) { try { context.getLogger().log("Processed message " + msg.getBody()); // TODO: Do interesting work based on the new message } catch (Exception e) { context.getLogger().log("An error occurred"); throw e; } } }
JavaScript
SDK for JavaScript (v2)
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using JavaScript.

exports.handler = async (event, context) => { for (const message of event.Records) { await processMessageAsync(message); } console.info("done"); }; async function processMessageAsync(message) { try { console.log(`Processed message ${message.body}`); // TODO: Do interesting work based on the new message await Promise.resolve(1); //Placeholder for actual async work } catch (err) { console.error("An error occurred"); throw err; } }

Consuming an SQS event with Lambda using TypeScript.

import { SQSEvent, Context, SQSHandler, SQSRecord } from "aws-lambda"; export const functionHandler: SQSHandler = async ( event: SQSEvent, context: Context ): Promise<void> => { for (const message of event.Records) { await processMessageAsync(message); } console.info("done"); }; async function processMessageAsync(message: SQSRecord): Promise<any> { try { console.log(`Processed message ${message.body}`); // TODO: Do interesting work based on the new message await Promise.resolve(1); //Placeholder for actual async work } catch (err) { console.error("An error occurred"); throw err; } }
PHP
SDK for PHP
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using PHP.

<?php # using bref/bref and bref/logger for simplicity use Bref\Context\Context; use Bref\Event\InvalidLambdaEvent; use Bref\Event\Sqs\SqsEvent; use Bref\Event\Sqs\SqsHandler; use Bref\Logger\StderrLogger; require __DIR__ . '/vendor/autoload.php'; class Handler extends SqsHandler { private StderrLogger $logger; public function __construct(StderrLogger $logger) { $this->logger = $logger; } /** * @param SqsEvent $event * @param Context $context * @return void * @throws InvalidLambdaEvent */ public function handleSqs(SqsEvent $event, Context $context): void { try { foreach ($event->getRecords() as $record) { $body = $record->getBody(); // TODO: Do interesting work based on the new message } } catch (InvalidLambdaEvent $e) { $this->logger->error($e->getMessage()); throw $e; } } } $logger = new StderrLogger(); return new Handler($logger);
Python
SDK for Python (Boto3)
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using Python.

def lambda_handler(event, context): for message in event['Records']: process_message(message) print("done") def process_message(message): try: print(f"Processed message {message['body']}") # TODO: Do interesting work based on the new message except Exception as err: print("An error occurred") raise err
Ruby
SDK for Ruby
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using Ruby.

def lambda_handler(event:, context:) event['Records'].each do |message| process_message(message) end puts "done" end def process_message(message) begin puts "Processed message #{message['body']}" # TODO: Do interesting work based on the new message rescue StandardError => err puts "An error occurred" raise err end end
Rust
SDK for Rust
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming an SQS event with Lambda using Rust.

use aws_lambda_events::event::sqs::SqsEvent; use lambda_runtime::{run, service_fn, Error, LambdaEvent}; async fn function_handler(event: LambdaEvent<SqsEvent>) -> Result<(), Error> { event.payload.records.iter().for_each(|record| { // process the record tracing::info!("Message body: {}", record.body.as_deref().unwrap_or_default()) }); Ok(()) } #[tokio::main] async fn main() -> Result<(), Error> { tracing_subscriber::fmt() .with_max_level(tracing::Level::INFO) // disable printing the name of the module in every log line. .with_target(false) // disabling time is handy because CloudWatch will add the ingestion time. .without_time() .init(); run(service_fn(function_handler)).await }
To create a Node.js Lambda function
  1. Create a directory for the project, and then switch to that directory.

    mkdir sqs-tutorial cd sqs-tutorial
  2. Copy the sample JavaScript code into a new file named index.js.

  3. Create a deployment package using the following zip command.

    zip function.zip index.js
  4. Create a Lambda function using the create-function AWS CLI command. For the role parameter, enter the ARN of the execution role that you created earlier.

    aws lambda create-function --function-name ProcessSQSRecord \ --zip-file fileb://function.zip --handler index.handler --runtime nodejs18.x \ --role arn:aws:iam::111122223333:role/lambda-sqs-role

Test the function


        Step 3 test the Lambda function

Invoke your Lambda function manually using the invoke AWS CLI command and a sample Amazon SQS event.

To invoke the Lambda function with a sample event
  1. Save the following JSON as a file named input.json. This JSON simulates an event that Amazon SQS might send to your Lambda function, where "body" contains the actual message from the queue. In this example, the message is "test".

    Example Amazon SQS event

    This is a test event—you don't need to change the message or the account number.

    { "Records": [ { "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d", "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...", "body": "test", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1545082649183", "SenderId": "AIDAIENQZJOLO23YVJ4VO", "ApproximateFirstReceiveTimestamp": "1545082649185" }, "messageAttributes": {}, "md5OfBody": "098f6bcd4621d373cade4e832627b4f6", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-1:111122223333:my-queue", "awsRegion": "us-east-1" } ] }
  2. Run the following invoke AWS CLI command. This command returns CloudWatch logs in the response. For more information about retrieving logs, see Accessing logs with the AWS CLI.

    aws lambda invoke --function-name ProcessSQSRecord --payload file://input.json out --log-type Tail \ --query 'LogResult' --output text --cli-binary-format raw-in-base64-out | base64 --decode

    The cli-binary-format option is required if you're using AWS CLI version 2. To make this the default setting, run aws configure set cli-binary-format raw-in-base64-out. For more information, see AWS CLI supported global command line options in the AWS Command Line Interface User Guide for Version 2.

  3. Find the INFO log in the response. This is where the Lambda function logs the message body. You should see logs that look like this:

    2023-09-11T22:45:04.271Z 348529ce-2211-4222-9099-59d07d837b60 INFO Processed message test 2023-09-11T22:45:04.288Z 348529ce-2211-4222-9099-59d07d837b60 INFO done

Create an Amazon SQS queue


        Step 4 create the Amazon SQS queue

Create an Amazon SQS queue that the Lambda function can use as an event source.

To create a queue
  1. Open the Amazon SQS console.

  2. Choose Create queue.

  3. Enter a name for the queue. Leave all other options at the default settings.

  4. Choose Create queue.

After creating the queue, note down its ARN. You need this in the next step when you associate the queue with your Lambda function.

Configure the event source


        Step 5 configure event source mapping

Connect the Amazon SQS queue to your Lambda function by creating an event source mapping. The event source mapping reads the Amazon SQS queue and invokes your Lambda function when a new message is added.

To create a mapping between your Amazon SQS queue and your Lambda function, use the create-event-source-mapping AWS CLI command. Example:

aws lambda create-event-source-mapping --function-name ProcessSQSRecord --batch-size 10 \ --event-source-arn arn:aws:sqs:us-east-1:111122223333:my-queue

To get a list of your event source mappings, use the list-event-source-mappings command. Example:

aws lambda list-event-source-mappings --function-name ProcessSQSRecord

Send a test message


        Step 6 send test message
To send an Amazon SQS message to the Lambda function
  1. Open the Amazon SQS console.

  2. Choose the queue that you created earlier.

  3. Choose Send and receive messages.

  4. Under Message body, enter a test message, such as "this is a test message."

  5. Choose Send message.

Lambda polls the queue for updates. When there is a new message, Lambda invokes your function with this new event data from the queue. If the function handler returns without exceptions, Lambda considers the message successfully processed and begins reading new messages in the queue. After successfully processing a message, Lambda automatically deletes it from the queue. If the handler throws an exception, Lambda considers the batch of messages not successfully processed, and Lambda invokes the function with the same batch of messages.

Check the CloudWatch logs


        Step 6 send test message
To confirm that the function processed the message
  1. Open the Functions page of the Lambda console.

  2. Choose the ProcessSQSRecord function.

  3. Choose Monitor.

  4. Choose View CloudWatch logs.

  5. In the CloudWatch console, choose the Log stream for the function.

  6. Find the INFO log. This is where the Lambda function logs the message body. You should see the message that you sent from the Amazon SQS queue. Example:

    2023-09-11T22:49:12.730Z b0c41e9c-0556-5a8b-af83-43e59efeec71 INFO Processed message this is a test message.

Clean up your resources

You can now delete the resources that you created for this tutorial, unless you want to retain them. By deleting AWS resources that you're no longer using, you prevent unnecessary charges to your AWS account.

To delete the execution role
  1. Open the Roles page of the IAM console.

  2. Select the execution role that you created.

  3. Choose Delete.

  4. Enter the name of the role in the text input field and choose Delete.

To delete the Lambda function
  1. Open the Functions page of the Lambda console.

  2. Select the function that you created.

  3. Choose Actions, Delete.

  4. Type delete in the text input field and choose Delete.

To delete the Amazon SQS queue
  1. Sign in to the AWS Management Console and open the Amazon SQS console at https://console.aws.amazon.com/sqs/.

  2. Select the queue you created.

  3. Choose Delete.

  4. Enter confirm in the text input field.

  5. Choose Delete.