Publish/Subscribe messaging, commonly known as pub/sub, is a communication pattern used primarily in serverless and microservices architectures. It enables asynchronous information exchange between distributed systems by decoupling the senders and receivers of messages.
One of the primary benefits of using a pub/sub messaging model is the decoupling it provides between components. Publishers and subscribers operate independently of each other, meaning they do not need to be aware of each other's existence or specific details. This decoupling allows for individual scaling of publishers and subscribers without affecting the overall system.
It also allows addition or removal of new publishers and subscribers without the need to modify existing components. Such flexibility is particularly essential for large-scale and dynamic network communication systems, where the ability to adapt to changing requirements is crucial.
How does pub sub messaging work?
Pub/sub systems have several main components that work together to enable asynchronous communication between distributed systems.
Here's a breakdown of all the key entities involved
1. Messages
Messages are the unit of data exchanged between publishers and subscribers in the pub/sub system. Messages can contain various types of data, such as text, JSON objects, or even binary data. They often include metadata like timestamps and topic information to provide context.
2. Message broker
The message broker is the central component of a pub/sub system. It acts as an intermediary between publishers and subscribers. Publishers send messages to the message broker, which then routes them to all the subscribers who are interested in the topic.
Message broker ensures efficient delivery and handles tasks like message queuing, filtering, and routing. There are two main types of message brokers:
Topic-based brokers
Route messages based on the topic they belong to
Content-based brokers
Route messages based on the content of the message itself.
3. Publishers
The publisher is the entity responsible for creating and sending messages to message brokers. Publishers categorize these messages into channels or topics based on their content.
4. Subscribers
The subscriber serves as the designated recipient of messages published to the message broker. By expressing interest in specific topics, subscribers ensure that they only receive messages that match their subscribed topics.
Pub/sub architecture with example
01.
A publisher initiates a message containing the necessary data or event information to be communicated.
02.
The publisher transmits this message to a specific topic on the message broker.
03.
The message broker receives the published message and identifies which subscribers have an interest in the topic of the message.
04.
The broker then routes and delivers the message to all subscribers who have expressed interest in (subscribed to) the topic.
05.
Subscribers receive the message from the broker and pro.
The weather app on your phone is a fantastic example of a pub/sub architecture in action. Let's break down the communication flow:
Publisher: A weather station acts as the publisher. It continuously generates and sends data about temperature (the message) to a designated topic named "weather-updates."
Topic: This "weather-updates" topic serves as a broadcasting channel specifically for weather-related information. It can include data like temperature, precipitation level, wind speed, and more.
Message broker: A central message broker receives messages from publishers (weather stations) about temperature updates. Identifies all the subscribers that have expressed interest in this topic and pushes the relevant data to them.
Subscriber: Your phone app receives the temperature data from the broker, since it is subscribed to the “weather-updates” topic and takes appropriate actions, such as displaying the information or sending out notifications based on the received data.
Unique features of a pub/sub messaging system over other communication architectures
1. Push Delivery
Unlike traditional message queuing where subscribers pull messages, pub/sub messaging uses a push model. When a message is published to a topic, the message broker actively pushes the message to all interested subscribers. This ensures real-time delivery and reduces latency.
2. Fanout
A key feature of pub/sub is fanout delivery. When a message is published to a topic, it gets delivered to all the subscribers registered for that topic. This is useful for scenarios where a single message needs to be broadcast to a large number of recipients.
3. Filtering
While fanout delivers messages to all subscribers, some systems allow for filtering. Subscribers can specify filters to only receive messages that meet certain criteria. This reduces unnecessary message processing for subscribers and improves efficiency.
Benefits of using pub sub messaging
1. Highly scalable
Systems can scale more easily because components (publishers and subscribers) can be added or removed without affecting each other. The message broker can balance the load among multiple nodes, making it possible to handle higher volumes of messages and larger numbers of clients.
2. Flexible and modular
The decoupling allows for flexible system reconfiguration. New subscribers can be added to topics dynamically, and publishers can be modified or replaced without disrupting the existing subscribers.
3. Improved fault tolerance
The system can better handle failures. If a publisher fails, subscribers are not directly affected and vice versa. The message broker can often provide persistence and replay capabilities to ensure that messages are not lost during failures.
4. Reduced complexity
Simplifies the communication logic. Publishers and subscribers only need to know about the topics they interact with, not the details of who is producing or consuming the messages. This reduces the overall complexity of the system design.
5. True asynchronous communication
Pub/sub messaging is inherently asynchronous. Publishers send messages to the message broker without waiting for a response from subscribers. This allows publishers to continue processing tasks without being blocked by subscriber availability. Similarly, subscribers receive messages at their own pace and can process them independently, leading to a more efficient use of system resources.
6. Improved Maintainability and Manageability
The centralized message broker simplifies system management. Monitoring message flow, identifying bottlenecks, and troubleshooting issues become easier compared to tightly coupled systems with complex communication channels.
Additionally, updates and changes to the message broker can impact the entire system, making upgrades and maintenance more efficient.
When to use pub sub architecture over other methods?
The pub/sub messaging pattern is particularly useful in scenarios that benefit from decoupled, asynchronous communication and require scalability, flexibility, and real-time data distribution.
Here are some specific situations where you can consider using pub/sub:
01.
When you want to minimize direct dependencies between components or services.
02.
When you need to deliver real-time updates or notifications to multiple clients or systems.
03.
When your system needs to handle a high volume of messages or events and scale horizontally.
04.
When implementing a microservices architecture where services need to react to events.
05.
When components need to communicate without waiting for immediate responses.
06.
When you need to send the same message to multiple subscribers simultaneously.
Limitations in using pub/sub messaging model
1. Overkill for simple systems
Pub/Sub can be excessive for straightforward systems that don't anticipate extensive scalability. If you're handling systems where scalability needs are static and not dynamic, reconsider the necessity of Pub/Sub. Likewise, for systems that need to interact with a limited number of other components or services, implementing Pub/Sub may be overly complex. It could be seen as extra baggage that isn't necessary.
2. Unreliable message delivery
Pub/Sub systems typically offer "at-least-once" or "once-but-not-once" delivery semantics. This means a message might be delivered multiple times to the same subscriber or potentially get dropped altogether. This can be problematic for scenarios requiring strict message processing and avoiding duplicates.
3. No delivery confirmation
Publishers typically don't receive confirmation if a message has been successfully delivered to all subscribers. They simply publish the message to the topic, and the broker takes care of routing it onwards. This lack of feedback can make it difficult to track delivery success and identify potential message loss.
4. Message ordering is not guaranteed
While some Pub/Sub systems offer features like message ordering keys, strict message ordering across all subscribers isn't always guaranteed. Messages might be delivered in a different order than they were published, especially under high load or with competing subscribers.
5. Limited resources affecting message delivery
Pub/Sub systems often have limitations on message size, retention periods, and message throughput. These limits can impact message delivery if publishers try to send too much data or if subscribers struggle to keep up with the message flow.
6. No guarantee of message delivery is the subscriber goes offline
Pub/Sub doesn't guarantee that a message will reach a subscriber if it's unavailable at the time of publishing. If a subscriber is offline or disconnected momentarily, it might miss the message entirely. Depending on the implementation, the message might be persisted for a certain period to allow the subscriber to catch up later, but there's no guarantee of retrieval
Alternatives to pub sub messaging
While Pub/Sub excels in real-time, low-latency communication, there are other options that can be equally good.
1. Message queuing
Message queuing is an alternate method to facilitate asynchronous communication. It acts as a reliable message broker, ensuring smooth information flow and decoupling components for better scalability and fault tolerance.
Advantages of using message queuing over pub-sub messaging:
Many Message Queues provide "exactly-once" delivery semantics. This ensures each message gets delivered only once to each subscriber, even in case of retries or failures.
Publishers can receive confirmation once a message has been successfully processed by the subscriber. This provides feedback on delivery success and helps identify issues.
MQs often offer robust message ordering. Messages are delivered to subscribers in the same order they were published, maintaining the sequence of events.
MQs typically persist messages until subscribers are available to receive them. This ensures messages aren't lost even if subscribers are temporarily offline.
Popular message queuing providers:
RabbitMQ: Open-source and feature-rich, offering message routing, retries, and message persistence.
Apache ActiveMQ: Another open-source option, known for its lightweight design and support for various messaging protocols.
Amazon SQS : A managed cloud-based solution offering scalable and cost-effective message queuing.
2. Websockets
While pub/sub & message queuing primarily focuses on asynchronous communication, websockets excel in real-time, bi-directional communication between web clients (browsers) and servers. They establish a persistent, open connection, enabling a continuous flow of data without page refreshes, ideal for building applications like in-app chat, collaborative editing, and live dashboards. However, it's important to understand that websockets aren't a direct replacement for Pub/Sub, but rather a complementary technology.
Advantages of using websockets over pub-sub messaging:
WebSockets establish a persistent connection, eliminating the need for constant requests and responses. Messages travel near-instantly, resulting in a more responsive and fluid user experience.
Messages are sent directly between connected parties, providing a more reliable delivery mechanism. However, it doesn't guarantee delivery if the client is disconnected.
The two-way nature of WebSockets allows for sending acknowledgement messages. The server can send confirmation messages back to the client upon successful message receipt.
Ordering is generally maintained for messages sent within a single WebSocket connection. In case of multiple clients, there's no guarantee of global message order across all connections.
The direct connection between client and server in WebSockets minimizes message transmission delays, leading to lower latency compared to Pub/Sub, which involves message routing through a central broker.
3. Chat APIs from communication infrastructure providers
If you are considering building a chat application from the ground up, using pub/sub messaging as the foundational layer for communication, it's important to note that while this approach offers flexibility, it also presents operational challenges. These challenges may require implementation of more complex systems, resulting in a significant investment of development time and resources.
To achieve a truly seamless chat experience, it is worth exploring the option of leveraging Chat APIs provided by communication infrastructure providers.
Chat APIs often come bundled with Chat SDKs – pre-built code libraries that streamline the integration process. These SDKs provide all the necessary tools to seamlessly integrate chat functionalities into your app, handling tasks like user interface elements, message rendering, and event handling.
Here's why opting for a Chat API like CometChat is the smarter way to go:
Chat APIs handle the heavy lifting – message delivery, presence management, user authentication – freeing you to focus on building the core features and user experience of your application. Pub/sub, while offering core functionality, requires additional development to implement these features.
With a Chat API, you can leverage pre-built features and functionalities, significantly reducing development time. This translates to getting your chat-enabled app to market faster and capitalizing on early user adoption.
Chat APIs come with a comprehensive set of features beyond basic messaging. Think one-on-one chats, group chats, file sharing, push notifications, and more – all readily available for integration.
Chat APIs are built for scalability and reliability, ensuring your app can handle growing user bases without performance issues. Pub/sub systems, while scalable, might require additional configuration and expertise to optimize for real-time chat applications.
Managing a pub/sub system can be a constant task. Chat APIs handle updates, bug fixes, and security patches, allowing you to focus on maintaining and improving your core application
Aarathy Sundaresan
Content Marketer , CometChat