Design a Metrics & Monitoring System

One server has run out of ingest rate. A fleet of machines is each exposing thousands of series, scraped every few seconds, and the total write rate is far past what one machine can pull in and append. The firehose is real now. But notice the shape of the problem before reaching for anything fancy: samples are small, they arrive roughly in time order, and once written they are never updated. That shape decides the write path, and it is the first real choice.

The obvious first answer is a general-purpose database — the kind most systems already run, a relational store on a B-tree. It is familiar, it is durable, and it can certainly hold rows of a series, a timestamp, and a value. The attraction is real: you know it, it queries flexibly, and you do not have to reach for something specialized. But watch what a firehose does to it. A B-tree keeps its data sorted in place, so every insert finds the right spot in the tree and writes it there, and under a relentless stream of inserts the tree is constantly being updated in place, splitting pages and rewriting them. Each tiny sample turns into a much larger amount of disk work — a small write amplifies into a big one. At a few thousand samples a second this is invisible; at hundreds of thousands a second per machine it is the bottleneck. A store that updates its data in place is fighting the workload, because the workload is almost pure append and the store keeps paying to stay sorted in place.

So the requirement sharpens: the write path has to be built for append, not for in-place update. That is exactly what a write-optimized store gives you, and it is why it wins over a general-purpose one. The general-purpose store was the familiar answer, but it amplifies every sample under a firehose; the write-optimized store buffers in memory behind a sequential log and flushes to immutable blocks, nothing ever updated in place, which is exactly what a relentless append-only firehose wants.

A write-optimized, append-friendly store, sharded across many machines. The write path has to be built for append, not for in-place update. That is exactly what a write-optimized store gives you, and it is the single-server design's head-block-and-log shape grown up. Incoming samples land in an in-memory head block, cheap because it is just memory. A write-ahead log on disk makes that memory durable — every sample is appended to a sequential log first, and a sequential append is the cheapest write a disk does, so there is no in-place update and no page splitting on the hot path. Every couple of hours the head block is flushed to an immutable on-disk block covering that time span, and a background process later merges small blocks into bigger ones. Reads and writes never fight over the same in-place structure, because written blocks are immutable and new data only ever appends. On disk each sample costs only one to two bytes, because consecutive samples in a series barely differ and are stored as tiny deltas. This is the write pattern behind Prometheus's own storage and behind purpose-built time-series stores generally — a named, well-understood primitive this design commits to without re-deriving its mechanics here.

Now size the ingest tier, and this is the both-sizings habit, because a metrics store has to be sized two different ways that give two different answers, and you take the larger. The throughput sizing is how many machines it takes to ingest the firehose: one real instance appends around half a million samples a second at the top end, and a fleet emitting roughly twelve million samples a second therefore needs on the order of tens of ingest instances just to keep up with the write rate — fleet samples a second over per-instance append rate, and it shards cleanly by splitting the series across more instances. The capacity sizing — how many machines it takes to hold the data — is a different calculation in different units, series times bytes per sample times retention over per-node RAM, and at real retention it is the far larger number and the wall the next use case is entirely about. So hold it for one more paragraph. The point here is that these are separate calculations, throughput is what you size the ingest tier by, and it spreads with the series.

The ingest tier swallows the firehose. Now comes the wall the brief warned about, and it is the heart of this whole board. The firehose was loud but tractable — you sharded it. Keeping what you swallowed is the problem that no amount of sharding solves, because it is not a rate problem, it is a total-cost problem, and the total cost is a number that grows in a way ordinary intuition badly underestimates. Operators want history — months of it for trends, years of it for capacity planning — but storing every raw sample at fleet resolution for years is a cost that grows with retention and, far worse, multiplies with the label dimension.

The aha is that this is not a compression problem you solve with a cleverer encoder, and it is not a disk-budget problem you solve by buying more disk. It is a multiplicative-cost problem. The storage cost is series count times bytes per sample times retention, and cardinality multiplies the first term: series count is the product of each label's distinct values, so one unbounded label with a thousand distinct values turns five million series into five billion, and every byte figure with it. Compression shrinks the bytes-per-sample term about twelvefold; rolling up and tiering by age shrinks the effective retention cost about forty-eightfold on the five-year-coarse-versus-one-year-raw comparison; but the only things that beat a multiplicative cost are refusing to multiply it and paying less per unit of aged time.

So the design needs two independent levers, because the wall grows on two axes. Cardinality control keeps the series count from exploding — refuse unbounded labels, cap active series per tenant — so the whole cost is not multiplied by a careless dimension. And the rollup pipeline coarsens data as it ages and tiers it from the fast store down to cheap object storage, expiring raw once a durable rollup exists, so the retention cost grows sub-linearly instead of as retention times full resolution. Neither lever alone clears the wall; together they turn a petabyte-a-year impossibility into an affordable design.

Bound the labels, and roll data up as it ages across tiered storage. Do the arithmetic out loud, because it is the argument. Take an illustrative instance of five million series at a fifteen-second resolution, and Prometheus's rule of about one-and-a-half bytes per sample. Build it up from the bytes: a sample every fifteen seconds is about two million samples per series per year, and at one-and-a-half bytes each that is roughly three megabytes per series per year, so across five million series one year of raw is roughly sixteen terabytes. Now compare it to keeping the same five million series but rolled up to a coarse one-hour resolution and kept for five years — the exact policy Uber's M3 documents. Five years at one sample an hour is about three hundred and thirty gigabytes for those series, roughly forty-eight times cheaper than a single year of raw, for five times the time span. The coarse tier is not a rounding error against the raw tier; it is a different order of magnitude. That comparison alone — before cardinality even enters — is why nobody keeps years of data at raw resolution.

And then cardinality enters, and it does not add, it multiplies. Series count is the product of each label's distinct values. Attach one unbounded label with a thousand distinct values — a user ID is the classic offender — and five million series becomes five billion. Every number above is multiplied by a thousand. The one-year-of-raw baseline goes from sixteen terabytes to about sixteen petabytes for a single year. This is the wall. And the tempting escape hatches do not clear it. Compression is real and valuable — encoding each sample as a tiny delta is exactly why a sample costs one to two bytes on disk instead of sixteen, roughly a twelvefold shrink — but that shrink is already baked into the numbers above, so there is no second twelvefold to take: the wall stays at petabytes a year. Buying more disk does not scale against a cost that multiplies with every careless label. So the wall forces two independent levers, on two different axes, and a real design needs both: cardinality control to bound the label dimension so the series count cannot explode, and the rollup pipeline over tiered storage to coarsen and age data so the retention cost grows sub-linearly. In practice cardinality control means refusing to put unbounded things — user IDs, email addresses, request IDs — into labels, and, where many tenants share a store, capping the active series any one tenant can create so a single misconfigured service cannot multiply the whole cluster's series count.

The second lever attacks the retention axis: as blocks age, roll their samples up to coarser resolution. Raw stays raw for a couple of days, then gets rolled up to one sample every five minutes, then to one sample an hour for the long tail — Thanos does this on a deliberate delay, waiting until a block is older than forty hours for the first rollup and older than ten days for the second, because recent data is exactly what operators query at full detail. And you tier the data as it ages: recent, high-resolution blocks live in the fast write-optimized store, and aged, rolled-up blocks move to cheap object storage. Once a durable rolled-up copy of a span exists in cheap storage, the original raw blocks for that span can be expired, because the coarse copy is enough for the long-range queries that reach that far back. Here is the myth this pipeline is most often mistaken for, and it matters because getting it wrong hides the real mechanism. Rolling data up does not, by itself, save storage. Thanos's own documentation is blunt: keeping raw and five-minute and one-hour resolutions all at once actually costs about three times more than raw alone, because each downsampled resolution is extra blocks stored alongside the raw, not instead of it. The purpose of the rollup is not to shrink disk — it is to make a query over months or years fast, by letting that query read a few coarse samples instead of scanning millions of raw ones. The storage saving is a separate decision: expiring the raw tier after its short retention window, which is only safe because the durable rollup already exists.

The write side is built: the firehose is sharded into the write-optimized store, cardinality is bounded, and aged data is rolled up and tiered to cheap storage. Now the operator opens a dashboard. All that storing was in service of this moment, and the read path has its own shape, forced by two facts about how metrics are actually queried. The first is recency skew, and it is dramatic: the overwhelming majority of queries are about the recent past. The second is about the rare query that reaches back months or years, and it is a safety fact rather than a speed one — a single query that pulls too many series or too long a range can exhaust one machine and fall over.

The naive read path treats every time range the same: take the query's range, scan the samples in it, aggregate, return. It is simple and uniform, and for a recent query it is even fine. But it is optimized for the wrong thing, because queries are not uniform in range — eighty-five percent of them read only the last twenty-six hours, and the rare one that reaches back years scans a vastly larger span. Treating all ranges equally means either serving the common recent query off disk when it could be in memory, or letting the rare long-range query scan millions of raw points and exhaust a machine. The read path is not one uniform scan; it is shaped by the skew.

So the requirement sharpens on two axes. Serve the common recent query from an in-memory hot tier, because that is where the queries actually are and a hot tier wins about seventyfold on latency for that mix. Run the rare long-range query over the coarse rolled-up tier, not over raw, because a few thousand coarse points is both fast and safe where millions of raw points is neither. And bound every query — a per-query memory ceiling, lazy decompression, cancellation on client disconnect — so no single query, however careless, takes a node down. Recent from memory, long from rollups, and never let one query exhaust a machine.

A hot tier for the common recent query, rollups for the rare long range, and a defensive query engine. The first fact is recency skew, and it is dramatic. The overwhelming majority of queries are about the recent past — an operator watching a deploy, debugging an incident happening now, checking the last few hours. Facebook measured this precisely: eighty-five percent of all queries read only the last twenty-six hours of data. That is not a mild lean; it means the read path is optimized for the wrong thing if it treats all time ranges equally. Because recent data is queried so much more than old data, keeping the recent window in an in-memory hot tier — rather than reading it off disk every time — pays for itself many times over. Facebook's in-memory tier in front of its on-disk store measured about a seventyfold query-latency improvement for exactly this reason. So the read path's first move is to serve the common recent query from the hot Time-series store, in memory, at about a millisecond.

The second fact is about the rare query, the one that reaches back months or years, and it is a safety fact. A long-range query cannot run over raw samples, for two reasons that point the same way. It would be slow, because months of raw samples is millions of points per series to scan. And it would be dangerous, because pulling that many points into memory to aggregate them can exhaust a single machine. This is not hypothetical: Uber's query engine hit a hard per-query memory ceiling of three and a half gigabytes from queries spanning very large numbers of series, and when the engine degraded under that pressure, operators kept refreshing their dashboards, piling new queries on top of ones that had never been cancelled, compounding the overload until the engine was redesigned around it. So the long-range query runs over the rollups, not the raw — it reads the coarse one-hour tier from the cold store, a few thousand points instead of millions, both fast enough and small enough to be safe. And the engine itself has to be defensive: cap the memory a single query may use, decompress data lazily rather than all at once, and cancel a query the instant the client that asked for it disconnects.

Step back and look at what is built. The ingest tier shards the firehose into a write-optimized store, cardinality is bounded at ingest, the rollup pipeline coarsens and tiers aged data to cheap storage, and the read path serves recent queries from memory and long ranges from rollups without letting one query take a node down. It is complete for the happy path. Now the premise arrives already broken, the way a real incident does — and the twist is that this is not an off-to-the-side failure that happens to touch the store. This is the wall this whole board was built around, hitting a real service in production, on the cardinality axis, before the retention math even gets a chance to bite. On the eleventh of March, 2021, Grafana Cloud's hosted Prometheus service had a roughly two-hour outage. It was not a flood of queries and not a disk filling up. It was a runaway series count.

The instinct, in the room and in the moment, is to read an out-of-memory cascade as a capacity shortfall and reach for the obvious lever — the cluster ran out of memory, so add more memory, raise the limit, give it room. And it is exactly the wrong move, for exactly the reason the signature dive established. The problem was never aggregate capacity. The problem was that one tenant's series count was unbounded, and an unbounded series count multiplies until it exhausts whatever memory you give it. Raising the limit is what let the tenant through in the first place — the forecast-based limit had been raised for their expected usage, and then the enforcement bug let them blow past even that. More memory would have bought a little time and then filled up too, because a multiplicative explosion outruns a linear addition of capacity.

The incident's deeper lesson is that cardinality control is only as good as its enforcement, and enforcement is easy to get subtly wrong. The limit existed; the bug was in applying it, so a tenant that should have been throttled at the edge was let through. A cardinality limit that is not actually enforced on the hot path is not a limit at all — it is a comment. Which is why the fix was not just lower the number but layers of ingestion-side protection, so that no single point of enforcement failing lets a runaway series count reach the store. The mechanism of bounding cardinality is simple to state; enforcing it reliably at ingest, as the workload shifts and new tenants arrive, is the ongoing engineering, and it is precisely where this incident went wrong.

Bound the cardinality — stricter per-tenant limits, correctly enforced, plus layered ingestion-side protection. A newly onboarded tenant started sending far more active series than its forecast-based limit had planned for — exactly the cardinality explosion this board's signature is about, a tenant's label dimension multiplying its series count past what anyone expected. That alone should have been caught by the per-tenant limit. But there was a bug in how the limit was enforced, so the runaway series growth was not stopped at the edge — it was allowed through, and it overwhelmed the cluster. The load drove a cascade of out-of-memory failures across the stack: the ingestion path ran out of memory under the series flood, and the failures spread outward until the internal auth gateways were failing their health checks and load balancers were pulling them from the pool. A single tenant's series count, unbounded by the broken limit, took the service down for about two hours.

The mitigation that actually resolved the incident was cardinality control, done right: stricter per-tenant active-series limits, enforced correctly this time, plus several added layers of ingestion-side protection so a runaway series count is stopped at the edge before it can overwhelm the store. That is the same lever this design already put on the Collector in the signature dive — cap the active series a tenant can create, and refuse the labels that explode the count. The reframe to carry out of this incident is the one the whole board is built on. Storage cost, and memory pressure, are multiplicative in the label dimension, and a multiplicative problem is not solved by adding a linear amount of hardware. You solve it by refusing to multiply — bounding cardinality at the source. Grafana learned it the expensive way, over two hours of outage; this design learns it as the signature. There is a read-path echo worth naming, because it shows the wall hits queries too: Uber's query engine hit its per-query memory ceiling precisely from queries that spanned enormous numbers of series — high cardinality on the read side — and the fix there was also to bound the blast radius, cap the query's memory and cancel it on disconnect, rather than give every query more memory. Ingest or query, the wall is the same shape: an unbounded count of series is a memory bomb, and the defense is bounding it, not feeding it.