Amazon SQS visibility timeout

When a consumer receives a message from an Amazon SQS queue, the message remains in the queue but becomes temporarily invisible to other consumers. This temporary invisibility is controlled by the visibility timeout, a mechanism that prevents other consumers from processing the same message while it is being worked on. Amazon SQS does not automatically delete the message; instead, the consumer must explicitly delete the message using the DeleteMessage action after it has been successfully processed.

Setting the visibility timeout

The visibility timeout in Amazon SQS begins as soon as a message is returned to a consumer. During this time, the consumer is expected to process and delete the message. However, if the consumer fails to delete the message before the visibility timeout expires, the message becomes visible again in the queue and can be retrieved by another consumer.

Every Amazon SQS queue has a default visibility timeout of 30 seconds, but you can adjust this setting based on your application's needs. It's typically best to set the visibility timeout to match the maximum time your application requires to process and delete a message. You can also configure a specific visibility timeout for individual messages without changing the queue’s overall timeout setting.

In flight messages and quotas

In Amazon SQS, in-flight messages are those that have been received from a queue by a consumer but have not yet been deleted. For standard queues, there is a limit on the number of in-flight messages, which can be a maximum of approximately 120,000, depending on queue traffic and message backlog.

• Short polling – If this limit is reached while using short polling, Amazon SQS will return an OverLimit error, indicating that no additional messages can be received until some in-flight messages are deleted.

• Long polling – If you are using long polling, Amazon SQS does not return an error when the in-flight message limit is reached. Instead, it will not return any new messages until the number of in-flight messages drops below the limit.

To manage in-flight messages effectively and avoid reaching this quota:

1. Prompt deletion – Ensure that messages are deleted as soon as they are successfully processed to reduce the in-flight count.

2. Monitor with CloudWatch – Use Amazon CloudWatch to monitor the number of in-flight messages and set alarms to alert you when approaching the limit.

3. Distribute load – Consider increasing the number of queues if you are processing a high volume of messages, to distribute the load and prevent bottlenecks.

4. Request a quota increase – If necessary, submit a support request to AWS to increase the in-flight message quota for your application.

Understanding visibility timeout in standard and FIFO queues

Visibility timeout in Amazon SQS manages message processing in both standard and FIFO queues to prevent duplicate processing and maintain message order.

• Standard queues – In standard queues, the visibility timeout helps prevent multiple consumers from processing the same message simultaneously. However, due to the at least once delivery model of Amazon SQS, there is no absolute guarantee that a message won't be delivered more than once during the visibility timeout period.

• FIFO queues – In FIFO (First-In-First-Out) queues, the visibility timeout ensures that messages with the same message group ID are processed in the correct order. If a consumer does not delete a message before the visibility timeout expires, no new messages with the same group ID are delivered until the message becomes visible again or is deleted. For FIFO queues, it is especially important to manage the visibility timeout carefully to maintain message order and processing integrity.

What happens if a consumer fails to process a message

If a consumer fails to process a message—due to issues such as application errors, crashes, or connectivity problems—and does not delete the message before the visibility timeout expires, the message automatically becomes visible in the queue again. Once visible, the message can be retrieved by the same or a different consumer for another processing attempt. This process ensures that messages are not lost even if the initial processing fails.

However, it's important to note that if the visibility timeout is set too high, it will delay the reappearance of unprocessed messages, potentially slowing down retries. Setting an appropriate visibility timeout based on the expected processing time is crucial for timely message handling.

Changing and terminating visibility timeout

Change or terminate the visibility timeout in Amazon SQS using the ChangeMessageVisibility action to manage message processing according to your needs.

• Changing the timeout – If the initial visibility timeout is insufficient, you can adjust it using the ChangeMessageVisibility action. This action allows you to either shorten or extend the timeout based on your processing needs.

• Terminating the timeout – If you decide not to process a received message, you can terminate its visibility timeout by setting the VisibilityTimeout to 0 seconds through the ChangeMessageVisibility action. This immediately makes the message available for other consumers to process.

Best practices

Use the following best practices for managing visibility timeouts in Amazon SQS, including setting, adjusting, and extending timeouts, as well as handling unprocessed messages using Dead-Letter Queues (DLQs).

• Setting and adjusting the timeout. Start by setting the visibility timeout to match the maximum time your application typically needs to process and delete a message. If you are unsure about the exact processing time, begin with a shorter timeout (for example, 2 minutes) and extend it as necessary. You can implement a heartbeat mechanism to periodically extend the visibility timeout, ensuring the message remains invisible until processing is complete. This minimizes delays in reprocessing unhandled messages and prevents premature visibility.

• Extending the timeout and handling the 12-Hour limit. If your processing time varies or may exceed the initially set timeout, use the ChangeMessageVisibility action to extend the visibility timeout while the message is being processed. Keep in mind that the visibility timeout has a maximum limit of 12 hours from when the message is first received. Extending the timeout doesn't reset this 12-hour limit. If your processing requires more time than this limit, consider using AWS Step Functions or breaking the task into smaller steps.

  In practice, setting the visibility timeout to the full 12-hour limit immediately may fail if there is a delay between receiving the message and updating the timeout. To avoid this, use a slightly lower value, such as 43,195 seconds.

• Handling unprocessed messages To manage messages that fail multiple processing attempts, configure a Dead-Letter Queue (DLQ). This ensures that messages that cannot be processed after several retries are captured separately for further analysis or handling, preventing them from repeatedly circulating in the main queue.