Where did the two seconds go?#
TL;DRthe 30-second version
- A trace is the record of one request's whole journey across every service it touched. A span is one unit of work inside that journey — one service handling one call — with a start time, an end time, and a pointer to its parent span.
- Spans nest into a tree. The root span is the whole request; each service it calls adds a child span; the tree shows you which hop was slow and which failed.
- For every service to stamp its spans with the same trace, the request carries a trace id and the current span id on every outbound call — usually in an HTTP header called `traceparent` (the W3C standard). This passing-along is called context propagation.
- You can't store a trace for every request at scale, so you sample: keep a fraction. Head-based sampling decides at the start and is cheap; tail-based decides after the request finishes and can keep the slow and failed ones, at a higher cost.
- OpenTelemetry is the vendor-neutral standard for producing traces; Jaeger, Zipkin, and Honeycomb are tools that store and show them.
Start with a backend split into many small services, each owning one job — a shape usually called microservices. A request to check out doesn't get handled in one place. The checkout service calls a pricing service, pricing calls the database, and checkout also calls auth to verify the user and inventory to reserve the item. One click can touch twenty services before the reply comes back. This spreading-out of one request into many downstream calls is called a fan-out.
Now the request is slow, and you want to know why. The obvious move is to open the logs. But each service writes its own logs, and each service only logged its own small part of the work. Checkout's log says it took 2200 milliseconds and returned fine. Pricing's log, in a different file on a different machine, says it did some work. Nothing connects the two. You can't tell that this line in pricing's log belongs to the same click as that line in checkout's log, because there's nothing shared between them to match on.
Follow one request across every service#
The fix starts with a single idea: give the whole request one shared id. When the request first arrives, generate a random id and call it the trace id. From then on, every piece of work done on behalf of that request gets stamped with that same trace id. Now checkout's work and pricing's work carry the same label, and you can pull every log line for the request by matching on one id. That id is what stitches the twenty services back into one story.
But a flat list of everything tagged with the trace id isn't enough. You need the shape: who called whom, and how long each call took. So each unit of work becomes a span. A span is one service doing one thing — handling the checkout call, or running the price query — and it records a start time, an end time (so you know its duration), a name, and the id of the span that caused it, called its parent. Checkout's span is the parent of pricing's span, because checkout called pricing.
Because every span points at its parent, the spans of one trace form a tree. The root span is the whole request; its children are the calls it made; their children are the calls those made. Lay the tree out on a shared timeline, with each span drawn as a bar from its start to its end, and the slow hop is impossible to miss — it's the long bar. Here's the trace for our two-second checkout:
- checkout · POST /buyroot span · 2200 ms total
- auth · verify-token5 ms
- pricing · GET /price2100 ms — the slow hop
- db · SELECT price2080 ms
- inventory · reserve15 ms
The tree answers the question at a glance. Auth took 5 ms, inventory took 15 ms, but pricing took 2100 ms — and almost all of that was a single database query at 2080 ms. The two seconds lived in one slow query, three hops deep, in a service you weren't even looking at. No amount of reading checkout's logs would have found it, because checkout only knew it called pricing and waited.
How does the id actually travel? On an HTTP or gRPC call (see /grpc), it rides in a request header. The industry standardized on one header so that tools from different vendors agree — the W3C `traceparent` header. It looks like this:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│ └──────────── trace id ─────────────┘ └── span id ──┘ │
version 32 hex chars (16 bytes) 16 hex (8 bytes) flags
version 00 the header format version
trace id the same on every span of this request — stitches them together
span id the caller's span — becomes the parent of the span the callee creates
flags 01 the 'sampled' bit: keep this trace. 00 means drop it.So on every outbound call, checkout sets `traceparent` with the trace id and its own current span id. Pricing reads the header, keeps the same trace id, and uses the incoming span id as its parent — which is exactly what builds the tree. The `traceparent` value changes at each hop (the span id is the caller's), but the trace id in the middle never changes for the whole request.
You cannot keep every trace#
Tracing looks free until you count the data. A trace isn't one record — it's one record per span, and a single request can easily produce twenty spans. Multiply that by your request rate and the numbers get large fast. Work it out for a mid-sized service.
PredictA service handles 10,000 requests per second. Each request produces about 20 spans, and each span is roughly 500 bytes once you include its name, timings, ids, and a few attributes. If you stored every span, how much trace data is that per day — and what does that force you to do?
Hint: Multiply requests × spans × bytes × seconds in a day. Then ask whether any budget survives that.
It's about 8.6 terabytes a day. Work it through: 10,000 requests/sec × 20 spans = 200,000 spans/sec. At 500 bytes each that's 100 MB/sec, which is 8.6 TB/day (100 MB × 86,400 seconds). That's for one service at a modest rate — a real fleet is many services at higher rates, so storing every span is quickly petabytes a month, plus the network cost of shipping it all and the query cost of searching it. You can't keep it all, so you keep a fraction. Deciding which traces to keep and which to drop is called sampling, and it's the central cost lever of a tracing system. Google's Dapper, the paper that started all this, sampled as aggressively as one trace in every 1,024 requests and still caught the problems that mattered.
The cheapest way to sample is to decide at the very start of the request: roll the dice when the request first arrives, and if it loses, don't record any spans for it at all. This is head-based sampling — the decision is made at the head of the request, before you know anything about how it went. The sampled bit in the `traceparent` header carries that decision downstream, so either the whole request is traced end-to-end or none of it is. It's cheap because unsampled requests cost almost nothing, and consistent because every service on the request agrees.
Head-based sampling has one painful flaw: it decides before it knows whether the request was interesting. The request that took two seconds, or the one that returned a 500 error, is exactly the one you wanted to keep — but the dice were rolled at the start, when it looked like every other request, so it was probably dropped along with the boring ones. The alternative that fixes this is tail-based sampling, covered in the trade-offs below.
Traces vs logs vs metrics
Tracing doesn't replace logs or metrics — it's the third of what's often called the three pillars of observability, and each answers a different question. Metrics are cheap aggregate numbers over time (requests per second, error rate, p99 latency); they tell you something is wrong but not which request or where. Logs are detailed per-event records inside one service; they tell you what happened but not how one request moved across services. Traces tie a single request's path together across every service. You reach for a different one depending on the question:
| Answers | Cost | Blind spot | |
|---|---|---|---|
| Metrics | Is anything wrong? (rates, error %, p99) | Tiny — pre-aggregated numbers | Can't point at one request or one hop |
| Logs | What happened inside this service? | Medium — one entry per event | Doesn't follow a request across services |
| Traces | Where did this one request spend its time? | High — many spans per request, so sampled | Sampling means most requests aren't kept |
In practice you use all three together: a metric alerts you that p99 latency spiked, a trace shows you the slow hop for an example request, and the logs on that hop's service tell you exactly what that call was doing. The strongest tools link them — a spike on a metrics chart can jump straight to an example trace behind it, a link called an exemplar (covered under the hood below).
Head-based vs tail-based sampling
The sampling choice is a real trade between cost and coverage, and it's a favorite thing to get wrong. Head-based decides at the start of the request and is cheap; tail-based decides at the end and is expensive but keeps the traces you actually want.
Tail-based sampling waits until the request has finished, then decides whether to keep it based on how it went — keep it if it was slow, keep it if it errored, otherwise keep a small random fraction. That's the ideal set of traces: you keep every problem and a representative sample of the healthy baseline. The catch is what it costs to do that. To decide at the tail, something has to hold all the spans of a request until the request is done — and because a request's spans are produced by different services on different machines, a central component (a collector) has to buffer them in memory and wait for the whole trace to arrive before it can judge it. That's memory, and it's the risk that a late or lost span makes the decision on an incomplete trace.
| Head-based | Tail-based | |
|---|---|---|
| Decides | At the request's start, before it runs | After the request finishes |
| Keeps the slow/failed ones? | Only by luck — decided before it knew | Yes — that's the point |
| Cost | Low — unsampled requests cost nothing | High — buffer every span until the trace is complete |
| Fails when | You need the rare error you dropped | A span is late/lost, or memory runs out |
The rule of thumb: start head-based because it's simple and cheap, and it's fine when you mostly want a representative sample. Move to tail-based when you specifically need every error and every slow request kept — usually once the system is big enough that the interesting requests are rare and getting dropped. Many production setups run a hybrid: a low head-based rate for baseline coverage, plus tail-based rules to guarantee the errors and outliers survive.
In the wild
- Google Dapper (2010) is where this all comes from — the internal system that traced Google's own services and defined the trace/span model, context propagation, and low-rate sampling everything since has followed. Its paper is still the clearest description of why and how.
- Zipkin came out of Twitter and was the first widely-used open-source tracer, modeled directly on Dapper; Jaeger came out of Uber and became the common open-source backend for storing and querying traces. Both take spans in and give you the tree-on-a-timeline view.
- OpenTelemetry (a merger of the earlier OpenTracing and OpenCensus projects, now a CNCF standard) is the instrumentation layer nearly everyone standardizes on today — it produces the spans and speaks the `traceparent` header, so you can send them to Jaeger, or to a vendor like Honeycomb, Datadog, or Grafana Tempo, without changing your code.
- Honeycomb built its product around high-cardinality tracing — keeping rich per-span attributes so you can slice traces by any dimension after the fact (which user, which build, which region) — and popularized tail-based sampling to keep the outlier traces that head-based sampling drops.
Pitfalls & gotchas
My trace just stops halfway — a service is missing from the tree. Why?
Almost always a broken context propagation. Somewhere along the chain a service made an outbound call without forwarding the `traceparent` header, so the next service saw no incoming trace id and started a brand-new trace (or none). The tree ends at the last service that still had the id. The usual culprit is a hop the tracing library doesn't auto-instrument — a raw HTTP client, or a boundary the library doesn't hook into.
The request goes through a queue. Does the trace survive that?
Only if you carry the context across the queue. When a producer puts a message on a queue (see /queue/sim) and a consumer picks it up later, there's no live connection between them to pass a header on. So you attach the trace id and span id to the message itself — in its metadata or headers — and the consumer reads them back to continue the trace. If you don't, the consumer's work shows up as a separate trace with no link to the request that caused it. Crossing an async boundary is the classic place propagation gets dropped.
A retried call shows up twice in the trace. Is that a bug?
No — that's correct, and it's useful. When a caller retries a failed request (see /idempotency), each attempt is its own real unit of work, so each becomes its own span, and they appear as sibling spans under the same parent. Seeing three sibling attempts where you expected one is exactly how a trace reveals that a downstream call was flapping and being retried — information a single log line would have hidden.
The child span looks like it started before its parent. How?
Clock skew. Each span is timestamped by the machine that produced it, and two machines' clocks are never perfectly in sync, so a child on a different host can appear to start a few milliseconds before its parent. Good trace viewers correct for this using the parent-child relationship rather than trusting raw timestamps, but it's why you don't read absolute cross-service timings down to the millisecond.
Go deeperUnder the hood: attributes, events, and exemplars
A span isn't just a name and two timestamps. It also carries attributes: key-value tags you attach to describe the work, like `http.status_code=500`, `user.id=42`, or `db.statement`. Attributes are what make a trace searchable after the fact — 'show me the slow traces where the region was eu-west' is an attribute filter. A span can also carry events: timestamped points inside the span's lifetime, like an exception being thrown, which pin an interesting moment to an exact time within the bar.
Attributes are also the bridge between the three pillars. An exemplar is a specific trace id attached to a metric data point — so a single dot on your p99 latency chart carries a pointer to one real trace that produced that number. You see the latency spike on the metric, click the exemplar, and land on an actual slow trace with its full span tree. That link is what turns 'p99 is high' into 'here's exactly which request was slow and why', without hunting for a matching trace by hand.
In an interview
Tracing comes up whenever a design has more than a couple of services, usually as 'how would you debug a latency problem across your services?' The strong answer isn't 'add logging' — it's the trace/span model plus how it survives crossing service boundaries. Hit these beats and name the traps:
- Lead with the problem: one request fans out across many services, and no single service's logs can tell you which hop was slow. That framing shows you know why tracing exists, not just what it is.
- Define the model cleanly: a trace is one request's whole journey; a span is one unit of work with a start, end, and parent; spans nest into a tree you read on a timeline. Name that the ids travel in the `traceparent` header — context propagation is the part interviewers probe.
- Name the sampling trade unprompted: you can't store every trace, so head-based sampling is cheap but drops the errors you wanted, and tail-based keeps the slow and failed ones at the cost of buffering spans centrally. Knowing both, and when to pick which, is the senior signal.
- Call out the critical path: the longest span isn't always the bottleneck if work runs in parallel — the request only waits on the critical chain. This distinction separates a real answer from a memorized one.
- Name the propagation traps: async boundaries (a queue) and retries. Say you'd carry the trace context in the message, and that retries correctly appear as sibling spans. Grounding it in OpenTelemetry and a backend like Jaeger shows you've actually used it.
PredictAn interviewer says: 'Your p99 latency alert is firing, but each service's own dashboards look healthy. What's going on, and what would you add?' What's the strong answer?
Hint: Which of the three pillars answers 'where across the path did this one request spend its time?' — and why can't the per-service dashboards see it?
Recognize that this is the exact gap tracing fills: metrics told you something is slow (the p99 alert), but per-service dashboards can't show it because the latency is spread across the path — each hop is individually fine, and it's the sum across the request that's slow, or one deep hop no single service's dashboard surfaces. The move is to add distributed tracing so you can pull an example slow trace and read its span tree to find the hop that dominates. Then go one level deeper: point at the critical path (the slow chain the total actually depended on, not just any long span), and note that to reliably catch these you'd want tail-based sampling or exemplars, because a rare slow request is exactly what head-based sampling tends to drop. Naming that metrics, logs, and traces answer different questions — and that this is a traces question — is what makes the answer strong.
References
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure (Google, 2010) — The origin paper — the trace/span model, context propagation, and low-rate sampling (1 in 1024).
- W3C Trace Context — The standard for the traceparent/tracestate headers that carry trace context between services.
- OpenTelemetry — Traces — The vendor-neutral standard's own explanation of traces, spans, and instrumentation.
- OpenTelemetry — Sampling — Head-based vs tail-based sampling, and how the sampled decision propagates.
- Jaeger — Architecture — A production open-source tracing backend: how spans are collected, stored, and queried.