Troubleshooting queue processing by Lambda functions - AWS Lambda

Troubleshooting queue processing by Lambda functions

This section uses SQS as an example but the principles apply to other services that queue messages for Lambda functions. The Lambda service processes messages from SQS in batches determined by the function’s batch size property. If there are still messages on the queue, Lambda then scales up the processing function up to an additional 60 instances per minute, to retrieve and process batches more quickly. In normal operation, this allows Lambda to maintain efficient processing of the queue, while scaling up and down in response to traffic.

This continues as long as the function is not producing errors, there is sufficient unreserved capacity in your account in the current Region (or there is remaining reserved capacity for the function), and there are messages available in the queue.

Service Default batch size Maximum batch size

Amazon Kinesis

100

10,000

Amazon DynamoDB Streams

100

1,000

Amazon SQS

10

10,000

Amazon MSK

100

10,000

To send batches of messages to SQS, this example uses the following code to generate messages from a Node.js script:

const AWS = require('aws-sdk') AWS.config.update({region: 'us-east-1'}) const sqs = new AWS.SQS() const BATCHES = 10 const sendMessage = async () => {   const params = {     Entries: [],     QueueUrl: 'https://sqs.us-east-1.amazonaws.com/1234123412/testQ'   }   for (let i = 0; i < 10; i++ ) {     const Id = parseInt(Math.random()*100000).toString()     params.Entries.push ({         Id,          MessageBody: `Message payload: ${Id}`     })   }   await sqs.sendMessageBatch(params).promise()  } const main = async() => {   for (let i = 0; i < BATCHES; i++ ) {     await sendMessage()   } } main().catch(error => console.error(error))

In this example:

  • An event producer is adding 10 messages to an SQS queue every second.

  • The AWS account only has unreserved concurrency of 100.