Design a Notification System

The single-server version welded the business service straight to the gateway, and that weld put a best-effort black box on the business critical path. Break it. Put a queue between the event and the send: when a business event happens, the App service drops a small notification event onto the queue and returns, and a separate pool of channel workers reads the queue and does the sending. The business service is decoupled — it commits its fact, hands off, and never waits on a provider. That much is forced. What is not obvious, and shapes every later decision, is what the workers are now calling: three channels, each through an external provider, and every one of them a best-effort, rate-capped black box.

Take the three channels one at a time, because each teaches the same lesson from a different angle. Phone push, through APNs or FCM, is fast and cheap and the one people most misunderstand. The payload is tiny — a regular push caps at about four kilobytes — so you cannot send the actual content; you send an id and a short line of text, and the app fetches the real thing when the user taps it. And push is explicitly best-effort: Apple's documentation states outright that the delivery of a remote notification is not guaranteed, and tells you never to put anything in the payload that cannot be re-fetched. FCM behaves the same way — it throttles you, caps a project at about six hundred thousand messages a minute, roughly ten thousand a second, and even caps each individual device at a couple of hundred messages a minute, which most designs forget until a runaway loop notifies one user hundreds of times.

Email, through SES, has a different shape but the same lesson. There are two independent limits — a rolling daily quota and a per-second rate — and, the part that surprises people, the quota is charged per recipient, not per send. One email to a fifty-person channel counts as fifty against your quota, not one. That single fact reshapes how you think about volume, and it comes back in a moment. SMS, through Twilio, is the slowest and most rate-limited: send faster than your per-second limit and Twilio does not reject the extras, it queues them and holds a message for up to ten hours before giving up. A short code gets you up to about a hundred a second. SMS is where 'I sent it' and 'it arrived, eventually, or not' are furthest apart.

Three providers, three shapes, the same three properties underneath every one. Each gateway is best-effort: accepted is not delivered. Each is rate-capped, with a hard ceiling you cannot push past. And each takes only a thin payload, so the notification carries a reference, not the content. Those three properties are the premise the entire rest of the design is built to work around.

A queue, one worker pool per channel, thin payload, every gateway best-effort. Decouple with a queue, and run one worker pool per channel, each sending a thin payload and treating every gateway as best-effort. There is no cleverness to choose here — the decision is to respect what the gateways are rather than pretend they are reliable. One pool per channel, because the channels have wildly different rates and failure modes and should not share a fate: SMS being slow should not back up push. The payload is always an id and a short line of text, never the content, because the channel caps it at a few kilobytes and the push may never arrive. And nothing downstream is ever allowed to assume that a gateway accepting a send means a user received it.

Now size the send tier, because this is the first of two sizings and it is the one that forces a box to split. Thirty thousand pushes a second, against a push provider that accepts about ten thousand a second per project, does not fit through one provider. Thirty thousand divided by ten thousand is three, so the push send tier spreads across about three sender pools, each pushing through its own provider project and staying under its ceiling. The email and SMS lanes size the same way against their own, much lower, ceilings. The throughput of the gateways, not the size of any data, is what splits this tier. The workers themselves hold no durable state — everything that must survive a crash lives in the queue, not in a worker — so a worker is a stateless box you scale by adding more of them.

The last use case said the App service 'drops a notification event onto the queue.' But in the same breath it is committing the business fact — writing 'this payment succeeded' to its own database. Those are two separate systems, the database and the queue, and the brief required that the fact and the notification succeed or fail together. So the real question is whether a write to the database and a publish to the queue can be one atomic operation. They cannot, and the ways people try to fake it are the debate.

The first attempt is the obvious one: write to the database, then publish to the queue right after. Commit the payment, then send the event. The gap between those two steps is the problem. If the process crashes after the commit and before the publish — and processes crash — the payment is recorded and the notification never happens: the user was charged and never told, the exact failure the brief forbade. Reordering does not help — publish first, then commit, and a crash in the gap sends a notification for a payment that never went through. There is no order of two independent writes that is safe against a crash between them.

The second attempt is to reach for a distributed transaction — a two-phase commit spanning the database and the broker, with a coordinator that makes both systems prepare, then commit together. This genuinely gives atomicity. It is also heavy and rarely worth it: it needs a coordinator, it holds locks across two systems while it waits, and if the coordinator stalls, both systems are stuck mid-commit. Most brokers and most teams avoid it, and it is the thing people wrongly assume the real answer is a lightweight version of. It is not.

One myth to kill on the way past, because it is the most common wrong description of the pattern: the outbox is not a lightweight two-phase commit. It is the opposite move. There is no coordinator, no prepare phase, no cross-system agreement. Atomicity is achieved by writing to only one system transactionally — the outbox row lives in the same database as the business row — and then treating the publish as a separate, retryable step that happens afterward. Two-phase commit makes two systems agree in one moment; the outbox refuses to make them agree at all, and instead makes the publish a thing that can safely be retried until it sticks.

The transactional outbox — one local transaction, then a relay publishes. Give the App service's own database a second table — the outbox. When the App service commits the business fact, in the same local transaction it also writes a row to the outbox: an event to publish. One transaction, one database, so it is atomic for free — either both the business row and the outbox row commit, or neither does. Then a separate process, the relay, reads new rows out of the outbox and publishes them to the queue. The business write is now safe and atomic, and the risky part — talking to the queue — has been moved out of the business transaction entirely.

And here is the hinge of the whole design, so slow down on it. The relay reads an outbox row, publishes it to the queue, and marks the row published. What if it crashes after publishing but before it marks the row? On restart it sees an unpublished row and publishes it again. The relay is therefore at-least-once by construction — it will, sometimes, publish the same event more than once. This is not a defect to fix. It is the direct, unavoidable price of not using a distributed transaction: the outbox trades 'the notification might be lost' for 'the notification might be duplicated,' and duplication is the far better problem, because it can be caught downstream while loss cannot be recovered. The whole next use case is about catching it.

The relay is at-least-once — that was the honest cost of the outbox, and now it comes due. Because the relay can publish the same event twice, the queue can hand the same event to the workers twice; and even if the relay were perfect, the queue itself redelivers on a consumer rebalance. So watch it with no defense in place. An event lands: notify user U with a push. A worker picks it up, calls the gateway, and U's phone buzzes. Then something ordinary happens — the pool rebalances, or the worker's acknowledgement is lost, or the relay republished after a crash — and the same event is delivered again, to another worker, which also calls the gateway. One event, two buzzes.

The reflex fix is 'the queue should handle this — turn on its deduplication,' and it is worth naming because it is the single most common wrong answer, wrong in a precise, checkable way. A plain queue like Kafka is at-least-once and does no deduplication at all. Amazon SQS in its standard mode says the same in its own docs: you may receive a message more than once, so design your consumer to be idempotent. SQS does offer a FIFO mode that advertises exactly-once — but read the fine print: that guarantee holds only inside a five-minute deduplication window. A duplicate that arrives six minutes after the original is not caught. Notification retries can easily span longer than five minutes — a real outage takes longer than that to clear — so the queue's own dedup window is exactly the wrong length for this problem. No queue in this survey closes the duplicate for you across the timescales that matter. The fix cannot live in the broker.

So the deduplication has to be something the worker does, on its own, out of a key it can recompute. That is the whole move: the identity of the notification — event, channel, recipient — is knowable to the worker without asking the broker anything, so the worker can recognize a redelivered event as the same notification and refuse to send it twice. The broker never had the information to dedup correctly across the timescales that matter; the worker does.

An idempotency key, checked at the channel worker before every send. Give every notification a key that identifies it uniquely — a hash of the event id, the channel, and the recipient. Before a worker sends, it checks a shared store for that key. If the key is absent, the worker writes it and sends. If the key is already present, this exact notification has already gone out, so the worker drops it and does nothing. The first delivery of a redelivered event writes the key and buzzes; the second delivery finds the key and stays silent. The double arrow collapses to one. This is Stripe's idempotency pattern, applied one layer out from where Stripe applies it: the worker is the client of the gateway, the key guards the send, and a redelivered event is a 'retry' that finds its key and is dropped.

Name the property carefully, because the lock is strict about the words. The queue gives at-least-once delivery. The worker's idempotency key adds deduplication on top. At-least-once plus idempotent handling gives effectively-once: every notification is delivered, and duplicates are collapsed, so the user experiences it exactly once even though the machinery underneath delivered it one-or-more times. Do not call it exactly-once — no channel and no queue here actually offers that as a guarantee. Exactly-once is the experience the user gets; effectively-once is the honest name for how you got there, by building deduplication at the edge instead of assuming it in the middle. Now the sizing, which is the second of the two and lands on a different box than the first. Each key is small — a hash, a marker, an expiry, call it fifty bytes. Hold them for a one-hour window at thirty thousand a second and you store about a hundred and eight million keys, roughly five gigabytes, which fits comfortably in one node. So unlike the send tier, whose throughput forced three pools, the dedup store's capacity does not force a split at all — the two sizings pinned different boxes.

The board is built and it is correct: the fact and the notification are atomic, the event flows through the queue, and duplicates collapse at the worker. Before trusting it, try to break it — and the crack is the one thing the happy path glossed. A worker calls a gateway and the send fails: a 429 because the project is over its minute quota, a timeout, a transient provider error. The notification is not delivered and it is not lost; it is sitting in a worker's hands, failed. What does the worker do with it?

The first idea is retry immediately, and keep retrying. It is the worst option for two reasons. A gateway returns 429 precisely because it is over capacity, and hammering it with immediate retries piles more load onto the thing that is already saturated. And a notification to a device that is genuinely gone — an uninstalled app, a dead number — will fail forever, so 'keep retrying' is an infinite loop with no exit.

The second idea is exponential backoff: wait one second, then two, then four, doubling each time, so a struggling gateway gets increasing room to recover. This is most of the way there, and it is what most people say. But AWS names a specific trap in plain exponential backoff, and it is worth respecting: it causes a thundering herd. All the sends that failed at the same instant — say, the moment the gateway briefly went over quota — back off by the same schedule, so they all retry at the same instants, in synchronized waves. The gateway gets slammed at one second, recovers, gets slammed again at two seconds. The backoff spread them in time but kept them in lockstep.

The third idea is the fix AWS recommends, and it is the pick: full-jitter backoff, which keeps the growing window but randomizes within it so the synchronized herd becomes a smooth trickle. With an exit — a bounded attempt count feeding a dead-letter queue — it is both kind to a struggling gateway and finite for a send that will never land.

Full-jitter backoff, with a bounded-attempt exit to a DLQ. Retry with full-jitter backoff, and give it an exit. Instead of waiting exactly the backoff interval, wait a random amount of time between zero and that interval — sleep = random(0, min(cap, base·2^attempt)). The window still grows with each attempt, so a gateway that stays down gets exponentially more room, but randomizing within the window scatters the retries so they no longer arrive in synchronized waves. The herd becomes a trickle. And there has to be an exit for the send that will never succeed: after a bounded number of attempts — a count often called maxReceiveCount — the notification stops being retried and is moved to a dead-letter queue, the DLQ. The DLQ is not a retry queue; it is a holding pen for a send that has exhausted its retries, sitting there for a human or an automated process to inspect — a dead device token to prune, a systemic gateway problem to investigate — rather than being lost or looping.

Here is where the seam from the last use case bites, and it is the subtlety this whole use case turns on. The retry policy has a total budget: with full-jitter backoff capped high enough to survive a real outage, a notification might make its attempts spread across many minutes — longer, easily, than a five-minute dedup window. Recall the rule from the dedup store: the window must be at least as long as the retry budget. Now you can see the whole trap. The DLQ's attempt count has no sense of time; the dedup window is a duration with no sense of attempts. If a late retry fires after the idempotency key has expired, the dedup store has forgotten the notification, so the retry sends and the user gets a duplicate — the very thing the signature dive worked to prevent, undone by a backoff cap set without looking at the window. The backoff cap, the DLQ retention, and the dedup window are one decision made in three places, and they have to be sized together.

Step back and look at what is built. It is correct and complete for a large, ordinary system: atomic hand-off, effectively-once delivery, retries with an exit. Now the premise arrives already broken, the way a real incident does. This is Discord's 2016 problem, and it took their push pipeline down. A perfect storm hit at once — a community crossed twenty-five thousand concurrent users just as a wave of new communities burst onto the platform — and the push pipeline flooded. Pushes, in Discord's own words, arrived late or did not arrive at all. Everything downstream looked healthy; notifications simply stopped landing.

The instinct is to add capacity — spin up more workers, add more sender pools, push harder. It is the reflex that worked two use cases ago when the send tier could not keep up, and here it is exactly wrong. The bottleneck was not Discord's workers. It was inside Firebase: each XMPP connection to Firebase accepts no more than a hundred pending requests at a time. Once you are at that ceiling, a thousand eager workers do not help — Firebase still only accepts a hundred in flight per connection, and everything beyond that queues, backs up, and slows.

Worse, pushing harder into a gateway that is already at its ceiling makes it worse: requests pile up, latency climbs, and 'accepted' stops meaning anything at all, because an accepted request is now sitting in a backlog that may never drain. Adding capacity upstream of a fixed downstream ceiling just moves the pile-up around. The only response that actually helps is to feed the gateway at exactly the rate it can accept and hold the rest in a buffer you control.

Backpressure — pull at the gateway's ceiling and buffer the overflow. Feed the gateway at exactly its rate — backpressure, not capacity. Discord's fix was a two-stage pipeline built for demand-based flow control: a collector receives and buffers incoming push requests, and a pusher pulls from it, but pulls exactly a hundred at a time, matching Firebase's pending-request ceiling, and asks for the next batch only as Firebase acknowledges the previous one. The whole pipeline runs at the speed Firebase can actually accept, by construction. When traffic is normal, the buffer is essentially empty, because the pusher keeps up. When a flood hits, the buffer absorbs it and the pusher keeps feeding Firebase at its steady hundred — the system slows to the gateway's rate instead of collapsing. Discord reports that buffer only engages about once a month, in genuinely catastrophic situations. That is the signature of good backpressure: invisible almost always, and the thing that saves you on the worst day.

The reframe is the lesson to carry out. A channel gateway is a best-effort black box with a hard ceiling, and you cannot push faster than it will accept, so the correct response to a flood is to slow down to the gateway's rate, deliberately, and let a buffer hold the overflow — not to scale your own side into a wall. This closes the loop with the retry dive — raw retries into a saturated gateway are the same mistake in miniature, which is why the backoff had to actually back off — and with the very first use case: 'the gateway accepted it' never meant 'the user got it,' and here is the incident that proves it, with numbers. And a buffer is not infinite, so when a relentless flood finally fills it, the right move is to shed the lowest-priority work — drop the marketing lane before the transactional lane — so the notifications that matter still get through at the gateway's rate. That is where the priority lanes from the first use case earn their keep a second time.