Amazon Simple Queue Service (Amazon SQS) offers reliable and scalable hosted queues for storing messages as they travel between computers. By using Amazon SQS, you can move data between distributed components of your applications that perform different tasks without losing messages or requiring each component to be always available.
Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.
Using Amazon SQS, you can decouple the components of an application so they run independently, with Amazon SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the Amazon SQS API.
The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.
Amazon SQS ensures delivery of each message at least once, and supports multiple readers and writers interacting with the same queue. A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.
Amazon SQS is engineered to always be available and deliver messages. One of the resulting tradeoffs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important. If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.
Be sure to read about distributed queues, which will help you understand how to design an application that works correctly with Amazon SQS. For more information, see Properties of Distributed Queues.
There are three main actors in the overall system:
The components of your distributed system
Messages in the queues
In the following diagram, your system has several components that send messages to the queue and receive messages from the queue. The diagram shows that a single queue, which has its messages (labeled A-E), is redundantly saved across multiple Amazon SQS servers.
Amazon SQS provides the following major features:
Redundant infrastructure – Guarantees delivery of your messages at least once, highly concurrent access to messages, and high availability for sending and retrieving messages
Multiple writers and readers – Multiple parts of your system can send or receive messages at the same time
Amazon SQS locks the message during processing, keeping other parts of your system from processing the message simultaneously.
Configurable settings per queue – All of your queues don't have to be exactly alike
For example, one queue can be optimized for messages that require a longer processing time than others.
Variable message size – Your messages can be up to 262,144 bytes (256 KB) in size
For even larger messages, you can store the contents of the message using the Amazon Simple Storage Service (Amazon S3) or Amazon SimpleDB, and use Amazon SQS to hold a pointer to the Amazon S3 or Amazon SimpleDB object. Alternately, you can split the larger message into smaller ones.
Access control – You can control who can send messages to a queue, and who can receive messages from a queue
Delay Queues – A delay queue is one which the user sets a default delay on a queue such that delivery of all messages enqueued will be postponed for that duration of time. You can set the delay value when you create a queue with
CreateQueue, and you can update the value with
SetQueueAttributes. If you update the value, the new value
affects only messages enqueued after the update.