Thread Pools & Work Queues

Start here: why not a thread per task?#

The simplest way to run work in the background is to start a thread for each piece of it: a request arrives, you spawn a thread, it does the work, it exits. This reads cleanly and it works — right up until load arrives. A thread is not free. Each one reserves a chunk of address space for its stack (commonly around a megabyte) and takes a kernel scheduling slot, so creating and destroying one has real, repeated cost. Under a steady stream of short tasks you spend a large fraction of your time just making and tearing down threads instead of doing the work.

The second, worse failure is that a thread-per-task design has no ceiling. If tasks arrive faster than they finish — a traffic spike, a slow dependency, a burst of retries — the thread count climbs without limit. Ten thousand threads is gigabytes of stacks reserved and ten thousand things for the scheduler to juggle, so the machine spends its cores context-switching rather than computing. The system does not degrade gently; it thrashes and stops. (Why more threads stops helping, and the difference between concurrency and parallelism underneath all of this, is the concurrency topic — see /concurrency/sim.)

The mechanism: a queue and N workers#

A thread pool has two parts: a set of worker threads, and a task queue they share. You submit a task — a small unit of work, a function to run — and instead of executing it yourself, you put it on the queue. Each worker runs the same tiny loop forever: take the next task off the queue, run it to completion, come back for another. When the queue is empty, the workers block on it, sleeping at zero CPU cost until a task is enqueued and one of them is woken to take it.

Two things fall out of this shape for free. Because the workers are permanent, submitting a task costs an enqueue and maybe a wakeup — no thread creation on the hot path. And because there are exactly N workers, at most N tasks run at once no matter how many are submitted; the rest wait their turn in the queue. That fixed N is the pool's concurrency limit, and it is the number the whole topic is about. The queue itself is the same decoupling buffer the backpressure topic builds — a shock absorber between a fast producer and a fixed-rate consumer — so its behavior under overload (it grows, the wait grows with it) is covered there; see /backpressure/sim.

The sizing question: how many workers?#

There is no universal pool size — the right number depends on what the tasks spend their time doing. The clean split is CPU-bound versus I/O-bound work, and the two want very different pools.

• CPU-bound tasks (hashing, compression, parsing, computation) keep a core busy the whole time they run. More workers than cores does not help — the extra threads just take turns on the same cores, adding context-switch overhead for no throughput. So the target is roughly the number of cores, often core count plus one to keep a core busy during the brief moment a thread stalls on a cache miss or page fault.
• I/O-bound tasks (a database call, an HTTP request to another service, reading a file) spend most of their time blocked, holding a thread but no core. Here you want many more workers than cores, because while one worker waits on I/O its core is free for another worker that has actual computing to do. Size it to the core count and your cores sit idle waiting; size it larger and you keep them fed.

Brian Goetz put a formula on the I/O-bound case in Java Concurrency in Practice. To keep your cores at a target utilization, the number of threads is the core count, times the utilization you want, times one plus the ratio of time a task spends waiting to time it spends computing:

There is a second way to reach the number, and it is the same law the backpressure topic uses to size a queue: Little's Law. The number of tasks in flight equals the arrival rate times the time each one takes to service. So the workers you need to keep up with a steady load is roughly the request rate multiplied by the average service time. Two hundred requests a second that each take half a second of wall-clock time (mostly waiting) means about one hundred concurrent tasks in flight — so a pool of around a hundred workers, not the eight your cores suggest. This is the same L = λ × W you can play with in /numbers/sim, pointed at threads instead of a queue.

When the pool is full: bound the queue, then choose#

Sizing decides how many tasks run at once. It does not decide what happens when tasks arrive faster than the pool can finish them — and that moment is where thread pools are won or lost. The work has to go somewhere, and there are only three somewheres: it waits in the queue, it is turned away, or it makes the caller wait.

The most important and most-broken decision is the queue's bound. It is tempting to use an unbounded queue so no task is ever refused. That is a trap, and it is the same trap the backpressure topic warns about: an unbounded queue does not add workers, it just lets the backlog grow longer before anything pushes back. Under sustained overload the queue climbs forever, latency climbs with it (a task waits behind everything ahead of it), memory climbs, and the process is eventually OOM-killed — a hard crash instead of a clean refusal. An unbounded queue does not prevent failure; it hides overload until the failure is catastrophic.

So the queue must be bounded, and once it is bounded you have to answer the real question: when the queue is full and every worker is busy, what happens to the next task? This is the saturation policy, and Java's ThreadPoolExecutor names the standard choices as rejection handlers:

These are not exotic — they are the backpressure toolkit expressed as pool settings. Reject is load shedding at the pool's door. Caller-runs is backpressure: it makes the fast producer feel the slow consumer, exactly like a full TCP receive buffer forcing a sender to pause. Block is backpressure taken to its extreme. The one to avoid by default is silent discard, because a task that vanishes without a trace is the hardest kind of overload to debug. Which failure you prefer is the whole design choice, and the backpressure topic lays out that choose-how-to-fail decision in full — this section is that decision scoped to a thread pool.

Thread pool vs the single-threaded event loop#

There is a completely different way to handle many concurrent tasks that do a lot of waiting: instead of a pool of threads that each block on I/O, run one thread that never blocks and juggles thousands of in-flight operations by their readiness. That is the event loop — the model behind Redis, Node.js, and nginx — and it is worth holding next to the pool because they make opposite bets. (The event loop has its own topic and simulator, /eventloop/sim; this is only the contrast.)

The honest summary is that they win on different axes. A thread pool lets you write simple blocking code and uses every core, but it pays a whole thread per waiting task, so massive I/O concurrency gets expensive. An event loop holds huge numbers of idle connections almost for free, but one thread means one core and one careless CPU-heavy task stalls everyone. This is why real systems combine them: an event loop at the network front for cheap concurrency, and a thread pool behind it for the blocking or CPU-heavy work the loop must not run itself. Node's own runtime does exactly this — libuv keeps a small thread pool (default four threads) to run file and DNS calls that have no non-blocking form, so the single JavaScript thread never blocks.