Message Queues
A message queue sits between a system that produces work and a system that processes it. The producer drops a message in. The consumer pulls it out later. Neither needs to know about the other.
Decoupling Producers and Consumers
When a web server has to send an email, the user shouldn't wait for the SMTP handshake. Instead, the server pushes a message onto a queue and returns immediately. A worker picks up the message later and sends the email. The web server is decoupled from the email service: either side can be slow, restarted, or replaced without breaking the other.
This pattern shows up everywhere: video encoding, billing, notifications, search indexing, analytics.
Pub/Sub
In a basic queue, each message goes to exactly one consumer. In publish/subscribe, a message goes to many subscribers. Publishers send to a topic. Subscribers register interest in topics. The broker routes each message to all current subscribers.
Pub/sub is how event-driven systems fan out updates. A new order can simultaneously update inventory, send a confirmation email, and feed analytics.
Ordering and Backpressure
Queues are usually FIFO within a single partition, so order is preserved. With multiple partitions or consumers, only per-key order is guaranteed. Choose a partition key that matches what you actually need ordered, like the user ID.
If producers outpace consumers, the queue grows. Backpressure mechanisms slow producers, drop low-priority messages, or alert operators before memory or disk runs out.
Delivery Guarantees
- At-most-once: fastest, but messages can be lost
- At-least-once: messages always arrive, but duplicates are possible. Consumers must be idempotent
- Exactly-once: messages arrive once and only once. Hard to implement and usually expensive
Most production systems aim for at-least-once with idempotent consumers.
Try It Yourself
- Extend the example so multiple consumers share the same queue. Show that each message is processed exactly once.
- Add a topic with two subscribers and verify both receive every message.
- Simulate a slow consumer and write code that drops the oldest message when the queue exceeds N items.