Publish / Subscribe

The problem: one event, many independent reactions#

The naive version is a direct call. The order service calls billing, then calls analytics, then calls email. This breaks in three ways at once. It is slow, because the order service waits for all three before it can respond. It is fragile, because if the email service is down the whole order fails. And it is rigid, because adding a fourth consumer means editing and redeploying the order service.

What we actually want is for the order service to state a fact once, "order 4271 was placed," and be done. Whoever cares about that fact can react to it, in their own time, without the order service knowing they exist. If one reactor is slow or broken, the others should be unaffected, and the broken one should get another chance later rather than losing the event.

Publish/subscribe is the decoupling that gives you exactly this. A producer publishes messages to a named topic. Interested consumers attach a subscription to that topic. A broker in the middle copies every published message to every subscription and then tracks, per subscription, whether that copy has been handled yet. The producer and the consumers never reference each other. They only reference the topic.

The mental model: a topic is a loudspeaker, a subscription is a mailbox#

Picture the topic as a loudspeaker in a room. When the producer speaks, everyone with a mailbox on the wall gets a written copy dropped into their own box. The speaker does not wait for anyone to read it, and does not even look up to see who is in the room.

Each mailbox is a subscription. The copy sits in the mailbox until its owner picks it up, works through it, and signs a receipt. If the owner drops the copy or never signs, the broker eventually notices and drops a fresh copy in the same box. Crucially, one owner signing their receipt does nothing to anyone else's mailbox. Every mailbox has its own copy and its own pace. That independence is the whole point of the pattern, and it is what a plain shared queue cannot give you.

How it works: fan-out and the delivery lifecycle#

There are two mechanisms stacked on top of each other. The first is fan-out: turning one publish into one copy per subscription. The second is the delivery lifecycle: how the broker walks a single copy from arrival to done, with retries in between.

When the producer publishes a message, the broker assigns it an id, records it on the topic once, and then places an independent copy into every subscription's pending set. A subscription with nothing attached to it still gets its copy; a subscription added tomorrow simply starts receiving copies of messages published from then on. Fan-out is why the same event can drive billing, analytics, and email without any of them knowing about the others.

Now follow a single copy inside one subscription. It starts pending, meaning it has arrived but nobody is working on it yet. When a consumer pulls it, the copy becomes in-flight: delivered, but not yet confirmed. The broker starts an acknowledgement deadline the moment it hands the copy over. From here, three things can happen.

• The consumer finishes the work and acknowledges the copy. The broker marks that copy done and never delivers it to this subscription again. This closes the copy for one subscription only; the message stays open on every other subscription until each one acks its own copy.
• The consumer reports failure with a negative acknowledgement (a nack). The broker puts the copy back into the pending set for redelivery, and remembers that this copy has now been delivered once.
• The consumer says nothing at all, because it crashed or hung. When the acknowledgement deadline passes, the broker treats the silence exactly like a nack: it assumes the delivery failed and returns the copy to pending.

A redelivered copy goes around the loop again: pending, then in-flight on the next pull, then ack or fail. Every failure increments a delivery counter for that copy. Once the counter passes a configured retry budget, the broker stops retrying and moves the copy to a dead-letter queue, a side channel where poison messages wait for a human or a separate handler to inspect them. Without a dead-letter queue a permanently failing message would redeliver forever and block progress.

  publish
     |
     v
 [pending] <-------------------+
     |                         |  nack / timeout
     |  deliver                |  (budget left)
     v                         |
 [in-flight] --- ack ---> [acked]  (done, never redelivered here)
     |
     |  nack / timeout, budget spent
     v
 [dead-letter]  (parked for inspection)

The engine on the simulator page is exactly this state machine. Publish a message and watch a copy land in every subscription. Deliver one, let its ack deadline lapse, and watch the same copy get delivered a second time. Nack a copy until its budget runs out and watch it fall into dead-letter, while the other subscription drains cleanly beside it.

At-least-once delivery and why you must be idempotent#

Look closely at the redelivery rule and a consequence falls out. The broker redelivers whenever it is unsure a copy was handled, and a missed ack looks identical whether the consumer crashed before doing the work or crashed just after doing it but before sending the ack. The broker cannot tell those apart, so it errs on the side of delivering again. This is at-least-once delivery: a message is delivered one or more times, and under failures the count is sometimes more than one.

That means a consumer can genuinely process the same message twice. If the handler charges a credit card, a duplicate delivery is a duplicate charge. The fix is not to make the broker perfect, because a broker that guarantees exactly-once across crashes is either very expensive or subtly lying. The fix is to make the handler idempotent: handling the message twice must land the same result as handling it once. In practice you give each message a stable id and record the ids you have already applied, so a repeat is recognised and skipped.

The opposite setting exists too. At-most-once delivery acks the message before doing the work, so a crash loses it and it is never retried. That trades duplicates for lost messages, which is fine for a metric you can afford to drop and wrong for a payment. Most durable pub/sub systems default to at-least-once because losing a message is usually worse than handling one twice, and idempotency is a tractable thing to build.

What it costs: storage, amplification, and the ack window#

Pub/sub does not have an interesting big-O; its costs are about storage and multiplicity. The first cost is fan-out storage. The broker must track delivery state per subscription, so a single message with N subscriptions produces N pieces of bookkeeping, and each subscription retains its copy until it acks. A topic with ten subscriptions and one slow consumer keeps that consumer's entire backlog resident while the other nine are empty.

The second cost is delivery amplification. Every redelivery is another delivery event, so a subscription with flaky consumers can deliver far more times than the publish count. The gap between total deliveries and total publishes is a direct read on how much retrying is happening, and a large gap is a signal that a consumer is failing rather than that traffic is high.

The third cost is the acknowledgement window. While a copy is in-flight it is invisible to other pulls on that subscription, so a long ack deadline protects slow work from spurious redelivery but also lengthens how long a truly-crashed consumer's message sits stuck. A short deadline recovers from crashes fast but risks redelivering work that was merely slow. Tuning this window is the main knob operators actually turn.

Trade-offs: the dials you actually set#

• Delivery guarantee. At-least-once (duplicates possible, nothing lost) is the durable default. At-most-once (no duplicates, loss possible) suits droppable data. Exactly-once is offered by some systems within a bounded scope, but it is costly and never free across arbitrary side effects, so idempotency remains the honest answer.
• Push versus pull. In pull mode consumers ask for messages when they have capacity, which gives them natural backpressure. In push mode the broker delivers to an endpoint, which is simpler but can overwhelm a slow consumer unless the broker also throttles.
• Ack deadline length. Longer deadlines tolerate slow work and reduce spurious redelivery; shorter deadlines recover from crashes faster. Systems let you extend the deadline for a message you are still working on rather than force one fixed value.
• Retry budget and dead-lettering. A generous budget survives transient hiccups; a tight budget quarantines poison messages sooner. Set it with a dead-letter queue attached so exhausted messages are inspected, not silently dropped.
• Ordering. Strict per-key ordering limits how much you can process in parallel, because you cannot advance past a stuck message in that key. Most pub/sub systems make ordering opt-in for this reason.

How it compares: pub/sub, work queue, and log#

Three patterns are easy to confuse because all three move messages from producers to consumers. The difference is who receives a given message and how progress is tracked.

The log is the closest cousin and the sharpest contrast. A log tracks a group's progress with one number, the committed offset, which is a bookmark that only moves forward. That number cannot express "message 5 is done but message 4 is still being retried," because an offset commits a contiguous prefix. Pub/sub tracks each message's state separately, so out-of-order acks and per-message redelivery are natural. The log wins on raw throughput and replay; pub/sub wins on per-message delivery semantics and independent per-consumer retries. Kafka's own internals (its partitioned, replicated log) are covered separately; this page is deliberately the delivery-semantics layer that sits above such a log.

Summary#