Creating and configuring an Amazon SQS event source mapping - AWS Lambda

Creating and configuring an Amazon SQS event source mapping

To process Amazon SQS messages with Lambda, configure your queue with the appropriate settings, then create a Lambda event source mapping.

Configuring a queue to use with Lambda

If you don't already have an existing Amazon SQS queue, create one to serve as an event source for your Lambda function. Then configure the queue to allow sufficient time for your Lambda function to process each batch of events.

To allow your function time to process each batch of records, set the source queue's visibility timeout to at least six times the configuration timeout on your function. The extra time allows Lambda to retry if your function is throttled while processing a previous batch.

By default, if Lambda encounters an error at any point while processing a batch, all messages in that batch return to the queue. After the visibility timeout, the messages become visible to Lambda again. You can configure your event source mapping to use partial batch responses to return only the failed messages back to the queue. In addition, if your function fails to process a message multiple times, Amazon SQS can send it to a dead-letter queue. We recommend setting the maxReceiveCount on your source queue's redrive policy to at least 5. This gives Lambda a few chances to retry before sending failed messages directly to the dead-letter queue.

Setting up Lambda execution role permissions

The AWSLambdaSQSQueueExecutionRole AWS managed policy includes the permissions that Lambda needs to read from your Amazon SQS queue. You can add this managed policy to your function's execution role.

Optionally, if you're using an encrypted queue, you also need to add the following permission to your execution role:

Creating an SQS event source mapping

Create an event source mapping to tell Lambda to send items from your queue to a Lambda function. You can create multiple event source mappings to process items from multiple queues with a single function. When Lambda invokes the target function, the event can contain multiple items, up to a configurable maximum batch size.

To configure your function to read from Amazon SQS, attach the AWSLambdaSQSQueueExecutionRole AWS managed policy to your execution role. Then, create an SQS event source mapping from the console using the following steps.

To add permissions and create a trigger
  1. Open the Functions page of the Lambda console.

  2. Choose the name of a function.

  3. Choose the Configuration tab, and then choose Permissions.

  4. Under Role name, choose the link to your execution role. This link opens the role in the IAM console.

    Link to execution role
  5. Choose Add permissions, and then choose Attach policies.

    Attach policies in IAM console
  6. In the search field, enter AWSLambdaSQSQueueExecutionRole. Add this policy to your execution role. This is an AWS managed policy that contains the permissions your function needs to read from an Amazon SQS queue. For more information about this policy, see AWSLambdaSQSQueueExecutionRole in the AWS Managed Policy Reference.

  7. Go back to your function in the Lambda console. Under Function overview, choose Add trigger.

    Function overview section of the Lambda console
  8. Choose a trigger type.

  9. Configure the required options, and then choose Add.

Lambda supports the following configuration options for Amazon SQS event sources:

SQS queue

The Amazon SQS queue to read records from.

Enable trigger

The status of the event source mapping. Enable trigger is selected by default.

Batch size

The maximum number of records to send to the function in each batch. For a standard queue, this can be up to 10,000 records. For a FIFO queue, the maximum is 10. For a batch size over 10, you must also set the batch window (MaximumBatchingWindowInSeconds) to at least 1 second.

Configure your function timeout to allow enough time to process an entire batch of items. If items take a long time to process, choose a smaller batch size. A large batch size can improve efficiency for workloads that are very fast or have a lot of overhead. If you configure reserved concurrency on your function, set a minimum of five concurrent executions to reduce the chance of throttling errors when Lambda invokes your function.

Lambda passes all of the records in the batch to the function in a single call, as long as the total size of the events doesn't exceed the invocation payload size quota for synchronous invocation (6 MB). Both Lambda and Amazon SQS generate metadata for each record. This additional metadata is counted towards the total payload size and can cause the total number of records sent in a batch to be lower than your configured batch size. The metadata fields that Amazon SQS sends can be variable in length. For more information about the Amazon SQS metadata fields, see the ReceiveMessage API operation documentation in the Amazon Simple Queue Service API Reference.

Batch window

The maximum amount of time to gather records before invoking the function, in seconds. This applies only to standard queues.

If you're using a batch window greater than 0 seconds, you must account for the increased processing time in your queue's visibility timeout. We recommend setting your queue's visibility timeout to six times your function timeout, plus the value of MaximumBatchingWindowInSeconds. This allows time for your Lambda function to process each batch of events and to retry in the event of a throttling error.

When messages become available, Lambda starts processing messages in batches. Lambda starts processing five batches at a time with five concurrent invocations of your function. If messages are still available, Lambda adds up to 300 more instances of your function a minute, up to a maximum of 1,000 function instances. To learn more about function scaling and concurrency, see Lambda function scaling.

To process more messages, you can optimize your Lambda function for higher throughput. For more information, see Understanding how AWS Lambda scales with Amazon SQS standard queues.

Maximum concurrency

The maximum number of concurrent functions that the event source can invoke. For more information, see Configuring maximum concurrency for Amazon SQS event sources.

Filter criteria

Add filter criteria to control which events Lambda sends to your function for processing. For more information, see Lambda event filtering.