The problem: two writes that can't happen together#
TL;DRthe 30-second version
- You need to change your database and publish an event about that change. They live in two different systems, so you can't do both in one atomic step.
- The fix: write the event as an ordinary row in an outbox table, inside the same database transaction as the business change. Now they commit together or not at all.
- A separate process, the relay, reads new outbox rows and publishes them to the broker — either by polling the table or by tailing the database's log.
- The relay guarantees at-least-once delivery: a crash can make it publish the same event twice. So the consumer dedups on the event's id.
- True exactly-once delivery is impossible, but exactly-once processing — every event acted on once — is achievable. That's the whole point.
Start with the everyday version of this. A service takes an order. It has two jobs: save the order to its database, and tell the rest of the company an order was placed. The first job is a database write. The second is publishing a message to a broker — a system like Kafka that carries events from the service that produced them to whatever services care.
The obvious code does them one after the other: save the order, then publish the event. This is a dual write — two writes to two separate systems in a row. And the moment there are two separate systems, there's a gap between them where the process can die. Power cut, deploy, a killed pod: pick your reason. If the service crashes in that gap, the order is safely in the database but the event was never published. The warehouse never hears about the order, and it silently never ships.
Flipping the order doesn't help. Publish first, then save, and now a crash in the gap gives you the opposite ghost: an OrderCreated event went out, downstream services believe an order exists, but the database write never happened, so there's no order behind it. Either way, one write lands and the other doesn't, and the two systems disagree about what's true.
Build it, one decision at a time#
The trouble is that the event has to travel to a second system before it's safely recorded. So don't send it to the second system yet. Write the event down in the one place you already commit atomically: your own database. Add a table called the outbox, and to publish an event, insert a row into it.
Now the two writes become one. In a single transaction, you insert the order row and insert one outbox row describing the OrderCreated event. Because they're in the same transaction and the same database, they commit together or roll back together. There's no gap: after the commit, either both rows exist or neither does. The event is now as durable as the order itself, and you never touched the broker to get that guarantee.
But an event sitting in a database table isn't published — the warehouse still can't see it. So add one more piece: a separate process, the relay, whose only job is to move committed outbox rows to the broker. It reads new rows from the outbox, publishes each one to the broker, and marks the row as sent. The order service's job ends at COMMIT; the relay picks up from there.
- Order arrives — BEGIN a transaction, INSERT the order row, INSERT an outbox row holding the event (a type, an id, and the JSON payload), COMMIT. Both land or neither does.
- Relay wakes up — it finds the new outbox row that hasn't been sent yet.
- Relay publishes — it sends the event to the broker's orders topic.
- Relay marks it done — it updates the outbox row (or deletes it) so it won't be sent again.
- Consumer reads — the warehouse service consumes OrderCreated from the topic and ships the order.
At-least-once, and how it becomes effectively-once#
Look closely at the relay's two steps: publish the event, then mark the row sent. There's a gap between them too — and this time we can't remove it, because they're again in two systems (the broker and the database). If the relay crashes after publishing but before marking the row, the row still looks unsent on restart. So the relay publishes it again. The event goes out twice.
You might try to close that gap by marking the row first, then publishing. But then a crash between them loses the event entirely — the row says sent, yet nothing was published. Given the choice between maybe-losing and maybe-duplicating, we pick duplicating: publish first, mark second. A duplicate can be cleaned up later; a lost event is gone. So the relay delivers every event at-least-once — never zero times, sometimes more than once.
This is a general truth, not a flaw in our relay. As long as the sender and the receiver acknowledge over a network that can drop the acknowledgement, the sender can never be sure a message arrived, so a safe sender must be willing to resend. Exactly-once delivery — every event put on the wire exactly one time — is impossible. What you can build instead is exactly-once processing: the event may arrive more than once, but its effect happens only once.
You get there on the consumer side. Give every event a unique id when you insert the outbox row (a UUID is fine). The consumer keeps a record of the ids it has already processed — a dedup store. When an event arrives, the consumer checks: have I seen this id? If yes, it's a duplicate, so skip it. If no, process it and record the id. Now the warehouse ships once even if OrderCreated is delivered three times. That id is an idempotency key, and this consumer-side dedup is covered in full on the idempotency page (/idempotency).
PredictThe relay publishes event 4471 to Kafka, then the pod is killed before it can mark the outbox row sent. When a new relay starts, what does the consumer see, and what stops the customer being charged twice?
Hint: What state is the outbox row in on restart, and what does the consumer do with an id it has seen before?
On restart the outbox row for 4471 still looks unsent, so the new relay publishes it again. The consumer now sees event 4471 twice. What saves it is the dedup store: the first delivery was processed and its id 4471 recorded, so when the second copy arrives the consumer looks up 4471, finds it already handled, and skips it. The customer is charged once. This is exactly the at-least-once-plus-dedup pairing — the relay is allowed to duplicate precisely because the consumer is built to ignore duplicates.
How the relay finds new events: polling vs log-tailing
The relay has to notice new outbox rows somehow. There are two ways, and they're a real design fork worth knowing by name.
The first is the polling publisher: the relay runs a query on a timer, asking the outbox table for rows that haven't been sent, for example every 200 milliseconds. It's a few lines of application code and needs nothing extra to run. The cost is that it queries the database over and over even when nothing new has arrived, and the delivery delay is up to one poll interval.
The second is transaction log tailing, also called change data capture. Instead of querying the table, a connector reads the database's write-ahead log — the same append-only log the database writes for its own crash recovery — and picks out the inserts into the outbox table as they're committed. It adds almost no load to the database because it reads the log rather than the tables, and it delivers within milliseconds of the commit. The write-ahead log is covered at /wal/sim, and change data capture as a stream at /cdc/sim.
Here's how the two compare.
| Polling publisher | Log tailing (CDC) | |
|---|---|---|
| How it finds new events | Queries the outbox table on a timer | Reads the database's write-ahead log |
| Latency | Up to one poll interval (e.g. 200 ms) | Milliseconds, as the row commits |
| Load on the database | Repeated queries, even when idle | Near zero — reads the log, not the table |
| Ordering | Order by an outbox sequence column | Log order is commit order, for free |
| What you run | A little app code | A connector like Debezium on Kafka Connect |
The log-tailing route is common enough that Debezium ships an Outbox Event Router built for exactly this: it captures inserts into your outbox table from the log and reshapes each one into a clean message on a per-type topic, using the event's type for the topic name and its aggregate id for the message key. AWS builds the same shape with DynamoDB Streams triggering a Lambda. Polling is the simpler default; teams reach for CDC when they want lower latency, lower database load, and free commit-order.
Under the hood: ordering, cleanup, and the inbox
Three details decide whether an outbox stays healthy in production.
Ordering. Some consumers care about the order of events for one entity — an order's Created must arrive before its Cancelled. Give the outbox a monotonically increasing sequence column (or lean on the log's commit order with CDC), and have the relay publish in that order. Then send all events for one order to the same broker partition, so their order is preserved on the way to the consumer. Events for different orders can still flow in parallel; you only need order within a single key.
Cleanup. Sent rows pile up, and the outbox is a live table your transactions write to, so you don't want it growing without bound. Do the arithmetic: at 5,000 events per second, an outbox you never prune grows by 5,000 × 86,400 ≈ 432 million rows a day. So either delete each row once it's confirmed sent, or mark it sent and let a background job delete rows older than a retention window. With CDC there's a nice trick: the connector only needs the log entry, not the row, so the relay can delete the outbox row in the very same transaction that inserts it — the row never really has to live, it exists just long enough to make the log entry.
The inbox. The dedup store on the consumer has a name too: the inbox pattern. The consumer records each processed event id in an inbox table, and ideally does that write in the same transaction as its own business change, so "I processed this event" and "I did the work" commit together. That closes the consumer's own dual-write gap — the mirror image of what the outbox does for the producer. Outbox on the way out, inbox on the way in.
When to reach for an outbox (and when not to)
Use an outbox whenever a state change in your database must reliably produce an event, and losing that event would leave the system inconsistent. That's most event-driven work: an order placed, a payment captured, a user signed up. If the event matters, the outbox is the boring, correct way to emit it.
- Good fit: a service that owns a relational (or any transactional) database and needs to publish domain events reliably, where you can tolerate at-least-once delivery and add dedup on the consumer.
- Weaker fit: the write side isn't a transactional database (a plain cache, an external API call), so there's no local transaction to piggyback the outbox row on — you need a different reliability story there.
- Overkill: an event nobody depends on for correctness, or work you can just do inline in the same request. Don't add a table and a relay to fire a best-effort metric.
- Watch the operational cost: an outbox adds a table, a relay to run and monitor, and consumer-side dedup. It's cheap insurance for events that matter and pure overhead for events that don't.
Outbox vs the alternatives
The outbox is one answer to the dual-write problem. Here's how it sits against the others you'll hear proposed.
| Approach | Atomic? | The catch |
|---|---|---|
| Publish after the commit | No | A crash after commit, before publish, loses the event |
| Publish inside the transaction | No | The transaction can still roll back after you've published — a phantom event |
| Two-phase commit across DB + broker | In theory | Most brokers don't support it; it blocks and is slow (see /twopc/sim) |
| Transactional outbox | Yes | At-least-once delivery, so consumers must dedup |
| Event sourcing | Yes | The event log becomes the source of truth — a much larger commitment |
Two-phase commit is the one people reach for first, because it promises true atomicity across the database and the broker. In practice brokers rarely support it well, and it's a blocking protocol that ties the commit's fate to every participant being available — the reasons it's largely avoided are laid out in the two-phase commit topic. The outbox sidesteps all of that by keeping the only atomic step inside a single database, and paying for it with dedup on the far end. It also composes with the saga pattern (/saga/sim): a saga coordinates a multi-step business transaction across services, and each step reliably emits its event through an outbox.
In the wild, and why it fits
The outbox is one of the most widely deployed reliability patterns in microservice systems, precisely because the dual-write problem shows up the instant a service both owns data and publishes events.
- Debezium (Red Hat) ships a dedicated Outbox Event Router: it tails Postgres, MySQL, or SQL Server logs and routes outbox inserts onto per-aggregate Kafka topics. This is the reference implementation of the log-tailing relay.
- Kafka Connect is the common runtime for that connector, so the relay is configuration rather than bespoke code — a big reason CDC-based outboxes are popular in Kafka shops.
- AWS builds the same shape natively: an outbox table in DynamoDB, DynamoDB Streams capturing inserts, and a Lambda publishing them to SNS or EventBridge — a serverless relay with no polling.
- The pattern is codified in Chris Richardson's microservices pattern language (transactional outbox, polling publisher, transaction log tailing, idempotent consumer) and taught in his book Microservices Patterns, which is why these exact names come up in interviews.
Common questions & gotchas
Why not just publish to Kafka right after the database commit?
Because the gap between the commit and the publish is exactly where the process can die. Commit succeeds, the pod is killed, and the event never goes out — the order exists but nobody was told. The outbox removes that gap by making the event part of the commit itself, and moving the publish to a relay that reads durable state.
Doesn't the relay have the same dual-write problem all over again?
It has the publish-then-mark gap, yes — but with a crucial difference. The relay reads from durable, committed state, so a crash never loses an event; it only risks sending one twice. That's a duplicate, which the consumer's dedup handles, not a loss. The outbox converts a can-lose problem into a can-duplicate problem, and duplicates are recoverable.
Can't I get true exactly-once delivery with the right broker?
No — exactly-once delivery over a network is impossible, because the sender can never be certain its message arrived (the acknowledgement can be lost) and so must be willing to resend. Kafka's "exactly-once" refers to exactly-once processing within a Kafka-to-Kafka pipeline using idempotent producers and transactions, not a guarantee that a message crosses a network exactly once. For an outbox, you get effectively-once by deduping on the consumer.
Should the relay poll or tail the log?
Start with polling — it's a few lines of code and needs nothing extra. Move to log tailing (CDC, e.g. Debezium) when you want lower latency, near-zero database load, and commit-order for free, and you're willing to run a connector. Both are legitimate; polling is the simpler default and CDC is the scale answer.
How do I keep the outbox table from growing forever?
Delete rows once they're confirmed sent, or mark them sent and prune with a background job on a retention window. With CDC you can go further and delete the row in the same transaction that inserts it, since the connector only needs the log entry — the row never has to persist at all.
QuizA team publishes events straight to Kafka right after their Postgres commit. Occasionally a downstream service is missing an event that clearly corresponds to a row that exists in the database. What's the most likely cause, and the standard fix?
- Kafka lost the message; enable more replicas and move on.
- The service crashed in the gap between the commit and the publish, so the event was never sent — fix it with a transactional outbox so the event commits with the row.
- The consumer is deduping too aggressively and dropping real events.
- The database transaction isn't isolated enough; raise the isolation level.
Show answer
The service crashed in the gap between the commit and the publish, so the event was never sent — fix it with a transactional outbox so the event commits with the row. — A row that exists with no matching event is the signature of a lost dual write: the commit landed, then the process died before publishing. More Kafka replicas don't help — the message was never produced. The fix is the transactional outbox: insert the event into an outbox table inside the same transaction as the row, so they're atomic, and let a relay publish it. (Dedup and isolation level are unrelated to an event that was never sent.)
In an interview
This comes up whenever a design says "and then we publish an event." The strong move is to name the dual-write problem before you're asked, then reach for the outbox — and, critically, to be honest that it buys you at-least-once, not exactly-once.
- State the problem first: writing to the database and publishing to a broker are two systems, so a crash between them loses or duplicates the event. There's no single transaction over both.
- Give the fix in one line: insert the event into an outbox table inside the same transaction as the business change, and have a relay publish committed rows to the broker.
- Be precise about the guarantee: the relay is at-least-once (it publishes then marks, so a crash can duplicate); the consumer dedups on the event id; together that's effectively-once processing. Say plainly that exactly-once delivery is impossible.
- Name the relay options: polling publisher (simple, a poll-interval of latency) vs transaction log tailing / CDC with Debezium (low latency, low DB load, commit-order). Knowing both signals depth.
- Contrast the alternative they'll fish for: two-phase commit across the DB and broker is atomic in theory but blocking, slow, and poorly supported — the outbox is why you don't need it.
PredictThe interviewer asks: "Your outbox relay just published a batch of 50 events, then crashed before recording that it did. It restarts and republishes all 50. Is your system broken?" What's the strong answer?
Hint: Where does correctness actually live — in the relay, or somewhere else?
No, and the reason is the point of the whole design. The relay is deliberately at-least-once: republishing after a crash is expected behaviour, not a bug. Correctness doesn't live in the relay — it lives in the consumer, which dedups on each event's id and processes each one only once. So the 50 republished events are recognized as already-seen and skipped, and no side effect fires twice. The strong answer names that the relay is allowed to duplicate exactly because the consumer is built to be idempotent, and that trying to make the relay itself exactly-once would be chasing an impossible guarantee.
References & further reading
- microservices.io — Pattern: Transactional outbox (Chris Richardson) — the canonical statement of the pattern and the dual-write problem
- microservices.io — Pattern: Polling publisher — the relay that queries the outbox table on a timer
- microservices.io — Pattern: Transaction log tailing — the CDC-based relay that reads the database log
- microservices.io — Pattern: Idempotent consumer — the consumer-side dedup that makes delivery effectively-once
- Debezium — Outbox Event Router — the reference log-tailing relay: outbox rows to per-aggregate Kafka topics