Replication: Sync, Async & Semi-sync

Start here: one machine is a single point of failure#

Put your whole database on one machine and you've built two failures into it. The first is obvious: that machine is a single point of failure, so when its disk dies at 3am, every read and every write stops until someone restores a backup — and a restore is often an hour of downtime, on data that was already hours stale. The second is quieter: one machine has a fixed amount of CPU and memory, so once reads saturate it, there's nowhere to put the next reader. You can't scale reads on a box that's already full.

The obvious first fix is a nightly backup copied to a second machine. It fails on both counts. A backup taken at midnight is useless for the writes that landed at 2am — they're simply gone — and promoting a cold backup to serve traffic takes long enough that the outage is real. What you actually need is a second machine that follows the first continuously, staying within milliseconds of it, ready to take over the instant the first one dies and ready to answer reads the whole time. That machine is a replica, and keeping it current is the problem this page is about.

How a copy stays current: ship the log#

The naive way to keep a copy current is to re-send the whole database every so often. On anything past a few gigabytes that's hopeless — you'd spend all your bandwidth copying rows that didn't change. So don't send the data; send the changes. A database already writes every change to an ordered, append-only file before it touches the real tables — the write-ahead log (WAL), the same log the /wal/sim topic builds. Each entry says exactly what changed: "set row 42's balance to 90", in order. That log is the perfect thing to replicate.

The mechanism is: the leader appends each change to its log and streams those entries to every follower; each follower replays them in the same order and lands in the same state. Because the log is ordered and every follower applies it in that order, the copies are deterministic — feed two followers the same log and they end up identical. This is replication, and log shipping is how nearly every single-leader system does it (PostgreSQL calls it streaming replication; MySQL ships its binary log; MongoDB ships an oplog). Raft, over in /raft/sim, is this same idea with an election bolted on to pick the leader safely.

The timing dial: sync, async, semi-sync#

Now the sharp question. A write lands on the leader. The leader appends it to its log and applies it locally. Before it tells the client "committed", does it wait for the followers to have the write too, or not? That single choice is the timing dial, and it trades durability against latency.

Asynchronous replication doesn't wait. The leader writes locally, acknowledges the client immediately, and streams the log to followers in the background. Writes are as fast as a single machine — no network round-trip on the write path. The cost hides in one word: the client was told "done" before any follower had the write. If the leader's disk dies in that gap, the write is gone, even though the client believes it succeeded. Acknowledged is not the same as safe.

Synchronous replication waits. The leader ships the write to a follower and holds the client's request open until the follower confirms it has the write on disk, then acknowledges. Now nothing acknowledged can be lost — there are always at least two copies of every committed write. The cost is also one thing: every single write now pays a network round-trip to the follower, and if that follower is slow or down, the write can't complete. Wait for all your followers and one sick replica stalls every write in the system.

Semi-synchronous replication is the middle ground, and it's what most durable setups actually run. The leader waits for one follower to confirm — not all of them — then acknowledges. You get the guarantee that every acknowledged write exists on at least two machines, but you only wait for the fastest follower to reply, not the slowest. One round-trip of safety, and a single slow replica doesn't stall you because the others can supply the confirmation.

The topology dial: who accepts a write#

The second dial is topology: how many machines are allowed to accept a write. There are three answers, and they climb in both power and difficulty.

Single-leader (also called primary-replica or primary-backup) is what we've described: one machine takes every write, the followers are read-only copies. Its great virtue is simplicity — writes are ordered by one machine, so there are never two conflicting versions of a row to reconcile. Every mainstream relational database defaults to this: PostgreSQL, MySQL, and MongoDB replica sets all have exactly one writable primary at a time. The limits are equally clear: all writes fund a single machine (you scale reads, not writes), and every write from a distant client crosses the network to that one leader.

Multi-leader (multi-master) lets more than one machine accept writes — typically one leader per datacenter, each streaming its log to the others. Now a user in Frankfurt writes to the Frankfurt leader at local speed, and if a whole datacenter goes offline the others keep taking writes. The price is severe: two leaders can accept conflicting writes to the same row at the same time — Frankfurt sets a field to "A" while Virginia sets it to "B" — and the system must later decide which wins. That conflict resolution is genuinely hard, and it's why multi-leader is reserved for cases that truly need multi-region writes. The /crdts topic is one principled way to make those concurrent writes merge without a lost update.

Leaderless (Dynamo-style) drops the leader entirely. The client sends each write to several replicas at once and reads from several at once. There's no promotion, no failover, no single writable node to lose. The trick that makes reads correct is the quorum rule from /quorum/sim: with N replicas, if every write lands on W of them and every read consults R of them, then setting W + R > N forces the read set and the write set to overlap by at least one replica — so a read always touches at least one copy that has the latest write. Cassandra, DynamoDB, and Riak work this way, letting you tune W and R per query to trade consistency against availability.

What it costs: replication lag and stale reads#

Every mode that doesn't wait synchronously shares one cost: a follower is always a little behind the leader. This gap is replication lag — the time between a write committing on the leader and that same write being applied on a follower. Under light load it's milliseconds. Under a burst of writes, or during a big batch update, a follower can fall seconds or even minutes behind while it works through the backlog. And whenever you read from a lagging follower, you get old data — a stale read.

Stale reads cause three specific, named anomalies. Each has a concrete fix, and interviewers ask for both.

• Read-after-write (also read-your-writes): you post a comment, the page reloads and reads a follower that hasn't got your write yet, and your own comment is missing — it looks like the write failed. The fix is to route a user's reads to the leader for a short window after they write, so people always see their own writes.
• Monotonic reads: you refresh, see a comment, refresh again and it's gone — because the two refreshes hit two different followers with different lag, and the second one was further behind. Time appeared to move backwards. The fix is to pin each user to one replica (by hashing their user id), so their reads never jump to a more-stale copy.
• Consistent-prefix reads: an observer sees an answer before the question, because two writes that had an order on the leader arrived on the follower out of order. The fix is to make sure causally-related writes go through the same partition, preserving their order.