Kafka Internals: the replicated log

Start here: the problem it solves#

Picture a single partition living on one broker. Producers append to it, consumers read from it, and life is simple until that broker's disk fails. Now every record on that partition is gone, and every consumer that was reading it is stuck. For a log that other systems treat as the source of truth, that's not an acceptable failure. So Kafka keeps copies.

The number of copies is the replication factor. A partition with replication factor 3 lives on three brokers. One of them is the leader: it handles every produce and every fetch for that partition. The other two are followers, and their only job is to stay identical to the leader. If a broker dies, another copy is ready to take over, and no data is lost. That's the whole point of replication.

But copies create a subtler problem than they solve. Suppose the leader accepts a write and immediately tells the producer "got it." A microsecond later the leader's disk dies before either follower copied that record. The record is gone, yet the producer was told it succeeded, and a consumer might already have read it. You've lost data you promised to keep, and worse, a reader acted on a message that no longer exists. The question replication forces you to answer is exactly this one: when is a write safe enough to acknowledge and safe enough to read?

The idea that almost works: read whatever the leader has#

Start with the simplest rule. The leader holds the newest, most complete copy of the log, so let consumers read anything the leader has written. A record is appended at offset 5, and instantly a consumer can read offset 5. It's fast and it feels right — why hide a record that's already on disk?

Watch it break. The leader appends offset 5 and a consumer reads it. Before either follower has fetched offset 5, the leader crashes. A follower is promoted to leader, but its log only goes up to offset 4 — it never got 5. Now the new leader has no offset 5, so if that consumer asks for offset 6 next, the log has effectively rewound underneath it. A record that was read is now missing. This is a phantom read, and it happens precisely because the leader let a record be read before the other copies had it.

The naive rule made one specific mistake: it treated "the leader has it" as "it's safe." But the leader is a single machine, and a record that lives on only one machine is one crash away from gone. Safety can't mean "one copy has it." It has to mean "enough copies have it that losing one is survivable."

The mechanism: followers, the ISR, and the high-water mark#

Every partition has one leader and a set of followers. The leader owns the log. Each follower runs a simple loop: ask the leader for records after the offset I last have, append what comes back, repeat. This is a pull model — followers fetch from the leader rather than the leader pushing to them — and it's the same fetch path consumers use, which keeps the system simple.

To talk about progress you need one number per replica: the log-end offset, or LEO. A replica's LEO is the offset it will write next, so it's just how many records that replica currently holds. The leader's LEO is always the furthest along, because the leader writes first. A follower's LEO trails the leader's by however far behind it is.

Now define the set that matters. The in-sync replica set, the ISR, is the group of replicas that are keeping up with the leader — the leader itself plus every follower that has fetched recently enough. A follower is in-sync as long as it has caught up to the leader within a time window; fall behind that window and you're out. The ISR is the set of copies Kafka currently trusts to hold the data.

The high-water mark is the payoff. It is the smallest LEO across the current ISR — in other words, the offset that every in-sync replica has reached. Records below the high-water mark are committed: every trusted copy has them, so any single broker can die without losing them. A consumer is only ever allowed to read below the high-water mark. That single rule is what makes a read durable. The leader may hold records above the mark, but they're written, not yet safe, and no consumer can see them.

The simulator runs exactly this. Load the first scenario, produce one record, and watch the leader append it while the high-water mark stays put. Each follower fetches in turn, and only on the final step — when every in-sync replica has the record — does the high-water mark advance and the record become readable. The gap between "the leader has it" and "you can read it" is the durability guarantee made visible.

This is also what a producer's acks setting controls. With acks=all, the leader waits to acknowledge a produce until the high-water mark has reached it, meaning the whole ISR has the record. With acks=1 the leader replies as soon as it alone has written the record, and with acks=0 it replies before even that. The three settings are a dial from fastest-and-riskiest to slowest-and-safest, and acks=all is the setting that actually gives you the guarantee this page is about.

When a follower falls behind#

The high-water mark waits for the slowest in-sync replica, and that cuts both ways. As long as every follower keeps up, the mark tracks the leader closely and writes commit quickly. But the day a follower slows down — a busy disk, a saturated network link, a long garbage-collection pause — it stops fetching, its LEO freezes, and because it's still in the ISR, it drags the high-water mark down with it.

Follow the consequence. The leader keeps appending, so its LEO climbs. The healthy follower keeps up. But the slow follower is frozen, and the high-water mark is the smallest LEO in the ISR, so the mark is pinned at the slow follower's frozen offset. New records pile up above the mark, written but unreadable. If producers are using acks=all, their writes now block, because the high-water mark isn't reaching the records they sent. One sick follower has stalled the whole partition's commits.

Kafka's answer is to stop waiting for a replica that isn't keeping up. Each follower has to catch up to the leader within a lag window — the replica.lag.time.max.ms setting, thirty seconds by default. A follower that falls behind that window is removed from the ISR by the leader. Once it's out, it no longer counts toward the high-water mark, so the mark is recomputed over the replicas that are keeping up and jumps forward. The backed-up records become readable, and acks=all producers unblock.

That jump forward has a cost, and it's worth saying plainly. The records that just became readable now live on fewer copies than before — the ISR shrank. Kafka let you keep making progress by lowering how many copies a committed record is guaranteed to be on. That's a deliberate trade of durability breadth for availability, and it's exactly the trade you tune with min.insync.replicas: if the ISR would shrink below that floor, acks=all writes are rejected instead, so you fail the write rather than commit it to too few copies.

When the leader dies: a clean handover#

Everything so far builds to the moment the leader itself fails. This is where the high-water mark earns its keep. When the leader broker goes down, the partition needs a new leader, and the rule for choosing one is strict: pick a replica from the ISR. An in-sync replica, by definition, has every record up to the high-water mark — every committed record. Promote one of those and no acknowledged data is lost. The controller, the component that manages leadership, does exactly this.

There's a second half to the handover, and it prevents a nasty failure. Imagine the old leader wasn't really dead — it was unreachable for a moment, and it comes back still believing it's the leader. If it kept accepting writes, you'd have two leaders diverging, which is a split brain. Kafka blocks this with the leader epoch: a version number for leadership that increases by one every time a new leader is elected. Every record is stamped with the epoch it was written under, and followers reject any request carrying an old epoch. When the recovered old leader tries to act, its epoch is stale, so it's refused. It's fenced.

The recovered broker then becomes a follower, discovers the higher epoch, and reconciles its log with the new leader — truncating any records it had written past the point the new leader agrees with, then fetching forward again. The epoch is also what lets it find that divergence point precisely. The result is one authoritative log, no lost committed records, and no two brokers accepting conflicting writes.

The third scenario runs the whole failover. Seed a few committed records so every replica is in-sync, then fail the leader. Watch the controller promote an in-sync follower, the leader epoch tick up by one, and the committed records survive intact on the new leader. The narration names the fence: the old leader's writes now carry a stale epoch and would be rejected.

What replication costs#

Durability isn't free, and the bill comes in three parts. The first is latency. With acks=all a produce isn't acknowledged until the high-water mark reaches it, which means at least one round trip to the slowest in-sync follower. That's the price of knowing the record survives a broker loss, and it's why acks=1 exists for workloads that would rather be fast than certain.

The second cost is storage and bandwidth. Replication factor 3 means every record is written three times, on three brokers, and shipped across the network to two followers. Triple the disk, and replication traffic that competes with client traffic for the same network. You size a Kafka cluster for the replicated write rate, not the raw produce rate.

The third cost is the tuning surface. Replication factor sets how many copies exist. The min.insync.replicas floor sets how many must acknowledge a write for acks=all to succeed. The lag window sets how patient the leader is before evicting a slow follower. A common durable configuration is replication factor 3 with min.insync.replicas 2: you keep three copies, require at least two to have a record before it commits, and can therefore lose any one broker without either losing data or blocking writes. Push min.insync.replicas to 3 and a single slow broker blocks every write; drop it to 1 and you're back to the phantom-read risk from the start of this page.

The trades you're actually making#

Each knob hands you one thing and quietly takes another. Naming the trade for each is what turns these settings from magic numbers into decisions.

The high-water mark trades latency for safety. By refusing to show a record until every in-sync replica has it, you guarantee a reader never sees data that a crash could erase. The cost is that a read-ready record waits for the slowest in-sync follower, so a lagging replica delays commits for everyone. You get durability and you give up speed.

Evicting a slow follower trades durability breadth for availability. Dropping a laggard from the ISR lets the high-water mark advance again, so writes keep flowing. But the records that commit while the ISR is shrunk live on fewer copies, so a second failure is now closer to losing data. You get progress and you give up some safety margin.

Requiring an in-sync leader trades availability for correctness. Insisting that only an ISR member can be promoted means a failover never loses committed data. The cost is that if no in-sync replica survives, the partition goes offline rather than serve a stale one. You get a log you can trust and you give up staying up in the worst case — unless you flip unclean leader election and make the opposite trade on purpose.

How this compares to other replication schemes#

Kafka's model is leader-based replication with a dynamic in-sync set, and it sits between two more familiar schemes. It helps to place it next to them.

The sharpest contrast is with a quorum system like Raft, which commits a record once a fixed majority — say two of three — has it. Kafka instead commits once the current in-sync set has it, and that set can shrink or grow. With min.insync.replicas set to 2 out of 3, the two schemes look almost identical in the common case. The difference is what happens under stress: a quorum always needs a majority, while Kafka can shrink its ISR to one replica and keep committing (giving up durability to stay available), or refuse to shrink past your floor and block instead. Kafka trades the rigidity of a fixed quorum for a knob you set per topic.

The consensus that actually elects leaders and tracks the ISR lives one layer down, in the controller. Classic Kafka kept that metadata in ZooKeeper; modern Kafka runs a built-in Raft-based controller quorum called KRaft. Either way, the partition data itself uses the leader-plus-ISR model on this page, and consensus is reserved for the small, critical job of agreeing who the leader is.