Publish-subscribe pattern

Intent Motivation Applicability Issues and considerations Implementation Blog references Related content

The publish-subscribe pattern, which is also known as the pub-sub pattern, is a messaging pattern that decouples a message sender (publisher) from interested receivers (subscribers). This pattern implements asynchronous communications by publishing messages or events through an intermediary known as a message broker or router (message infrastructure). The publish-subscribe pattern increases scalability and responsiveness for senders by offloading the responsibility of the message delivery to the message infrastructure, so the sender can focus on core message processing.

In distributed architectures, system components often need to provide information to other components as events take place within the system. The publish-subscribe pattern separates concerns so that applications can focus on their core capabilities while the message infrastructure handles communication responsibilities such as message routing and reliable delivery. The publish-subscribe pattern enables asynchronous messaging to decouple the publisher and subscribers. Publishers can also send messages without the knowledge of subscribers.

Use the publish-subscribe pattern when:

Parallel processing is required if a single message has different workflows.
Broadcasting messages to multiple subscribers and real-time responses from receivers aren't required.
The system or application can tolerate eventual consistency for data or state.
The application or component has to communicate with other applications or services that might use different languages, protocols, or platforms.

Subscriber availability: The publisher isn't aware whether the subscribers are listening, and they might not be. Published messages are transient in nature and can result in being dropped if the subscribers aren't available.
Message delivery guarantee: Typically, the publish-subscribe pattern can't guarantee the delivery of messages to all subscriber types, although certain services such as Amazon Simple Notification Service (Amazon SNS) can provide exactly-once delivery to some subscriber subsets.
Time to live (TTL): Messages have a lifetime and expire if they aren't processed within the time period. Consider adding the published messages to a queue so they can persist, and guarantee processing beyond the TTL period.
Message relevancy: Producers can set a time span for relevancy as part of the message data, and the message can be discarded after this date. Consider designing consumers to examine this information before you decide how to process the message.
Eventual consistency: There is a delay between the time the message is published and the time it's consumed by the subscriber. This might result in the subscriber data stores becoming eventually consistent when strong consistency is required. Eventual consistency might also be an issue when producers and consumers require near real time interaction.
Unidirectional communication: The publish-subscribe pattern is considered unidirectional. Applications that require bidirectional messaging with a return subscription channel should consider using a request-reply pattern if a synchronous response is required.
Message order: Message ordering isn't guaranteed. If consumers require ordered messages, we recommend that you use Amazon SNS FIFO topics to guarantee ordering.
Message duplication: Based on the messaging infrastructure, duplicate messages can be delivered to consumers. The consumers must be designed to be idempotent to handle duplicate message processing. Alternatively, use Amazon SNS FIFO topics to guarantee an exactly-once delivery.
Message filtering: Consumers are often interested only in a subset of the messages published by a producer. Provide mechanisms to allow subscribers to filter or narrow down the messages they receive by providing topics or content filters.
Message replay: Message replay capabilities might depend on the messaging infrastructure. You can also provide custom implementations depending on the use case.
Dead-letter queues: In a postal system, a dead-letter office is a facility for processing undeliverable mail. In pub/sub messaging, a dead-letter queue (DLQ) is a queue for messages that can't be delivered to a subscribed endpoint.

In a publish-subscribe pattern, the asynchronous messaging subsystem known as a message broker or router keeps track of subscriptions. When a producer publishes an event, the messaging infrastructure sends a message to each consumer. After a message is sent to subscribers, it is removed from the message infrastructure so it can't be replayed, and new subscribers do not see the event. Message brokers or routers decouple the event producer from message consumers by:

Providing an input channel for the producer to publish events that are packaged into messages, using a defined message format.
Creating an individual output channel per subscription. A subscription is the consumer's connection, where they listen for event messages that are associated with a specific input channel.
Copying messages from the input channel to the output channel for all consumers when the event is published.

Amazon SNS

Amazon SNS is a fully managed publisher-subscriber service that provides application-to-application (A2A) messaging to decouple distributed applications. It also provides application-to-person (A2P) messaging for sending SMS, email, and other push notifications.

Amazon SNS provides two types of topics: standard and first in, first out (FIFO).

Standard topics support an unlimited number of messages per second, and provide best-effort ordering and deduplication.
FIFO topics provide strict ordering and deduplication, and support up to 300 messages per second or 10 MB per second per FIFO topic (whichever comes first).

The following illustration shows how you can use Amazon SNS to implement the publish-subscribe pattern. After a user makes a payment, an SNS message is sent by the Payments Lambda function to the Payments SNS topic. This SNS topic has three subscribers. Each subscriber receives a copy of the message and processes it.

Amazon EventBridge

You can use Amazon EventBridge when you need more complex routing of messages from multiple producers across different protocols to subscribed consumers, or direct and fan-out subscriptions. EventBridge also supports content-based routing, filtering, sequencing, and splitting or aggregation. In the following illustration, EventBridge is used to build a version of the publish-subscribe pattern in which subscribers are defined by using event rules. After a user makes a payment, the Payments Lambda function sends a message to EventBridge by using the default event bus based on a custom schema that has three rules pointing to different targets. Each microservice processes the messages and performs the required actions.