Part 3 · Inference Efficiency Core

This is the heart of the book. Parts 1 and 2 built the physics: a datacenter GPU has far more compute than memory bandwidth, decode is sequential and memory-bound (see prefill vs decode), and the KV cache is what lets each new token avoid re-reading the whole prompt. Now we cash that understanding in. Every technique in this part is a direct, mechanical response to one of those bottlenecks — and together they are the difference between a model that “works in a notebook” and one that serves millions of requests at a price someone will actually pay.

Why this part is the heart

In 2022 you could load a 13B model and generate text, and it felt like magic until you tried to serve ten users at once. Latency ballooned, the GPU sat 80% idle, and your dollar-per-token made the whole thing unshippable. The throughline of this book — what does this cost in latency, memory, and dollars, and how do we make it cheaper? — has its sharpest answers right here. The techniques below took the same hardware and the same model and improved real-world throughput by an order of magnitude. No new chips, no smaller model: just refusing to waste the silicon you already paid for.

Each one attacks a specific kind of waste:

Waste	Symptom	Fix in this part
The KV cache eats HBM	batch size capped → low throughput	size it (KV math), then pack it (PagedAttention)
Requests finish at different times	GPU lanes sit idle mid-batch	continuous batching
Reserving max-length KV per request	most of HBM is empty padding	PagedAttention / vLLM
Attention writes an n×n matrix to HBM	memory-bandwidth-bound, O(n²) memory	FlashAttention
Decode does one token per weight-read	the GPU is starved per step	speculative decoding
Re-running prefill on shared text	wasted FLOPs and TTFT	prefix / prompt caching

The roadmap

Read these in order — each builds on the last.

1. KV Cache Math — the formula for the dominant memory cost. Until you can size the KV cache on paper, every later technique is hand-waving. We compute it for a real model and show why it, not the weights, caps your batch size.
2. Continuous Batching — stop making requests wait for the slowest one in their batch. Admit and evict at the token level so the GPU stays full.
3. PagedAttention & vLLM — treat the KV cache like OS virtual memory: fixed-size pages allocated on demand, near-zero waste, and copy-on-write sharing. The engine that made continuous batching practical.
4. FlashAttention — the same attention math with a fraction of the HBM traffic, by tiling through fast on-chip SRAM and never materializing the score matrix. Pure memory-movement win.
5. Speculative Decoding — a small draft model proposes several tokens; the big model verifies them all in one cheap parallel pass. Fewer sequential big-model steps, lossless output.
6. Prefix & Prompt Caching — many requests share a prefix (system prompt, few-shot block, a document). Cache its KV once and skip the redundant prefill forever after.

How these stack

These are not alternatives — production stacks use all of them at once. vLLM, for example, combines PagedAttention, continuous batching, FlashAttention kernels, and automatic prefix caching, and bolts speculative decoding on top. The multiplicative effect is why per-token prices have fallen so fast.

You still need no GPU

As before, every claim comes with a worked number you can check on paper. By the end of this part you will be able to estimate, for a given model and traffic pattern, how many concurrent requests a GPU can hold and roughly what each token costs.

The single sentence to carry through all six pages: decode is memory-bound, so the game is to read each weight from HBM as few times as possible and serve as many tokens as possible per read. Batching amortizes the read across requests; paging lets you batch more; FlashAttention cuts the reads attention needs; speculative decoding gets multiple tokens per read; prefix caching skips reads entirely. Same idea, six angles.

Check your understanding

1. What single hardware fact from Parts 1–2 makes every technique in this part necessary?
2. In one sentence each, name the waste that continuous batching and PagedAttention respectively eliminate.
3. Why are these techniques described as multiplicative rather than mutually exclusive?
4. Restate the “one sentence to carry through” in your own words.
5. Which technique skips computation entirely rather than making it cheaper, and what condition must hold for it to apply?

Show answers

1. Decode is sequential and memory-bandwidth-bound: the GPU spends each step reading weights from HBM, so it sits idle unless we serve many tokens per read.
2. Continuous batching eliminates idle GPU lanes caused by requests in a batch finishing at different times. PagedAttention eliminates the wasted HBM from reserving a contiguous max-length KV region per request.
3. They attack different wastes and compose: a real serving stack (e.g. vLLM) runs PagedAttention, continuous batching, FlashAttention, and prefix caching simultaneously, so their gains multiply.
4. Decode is memory-bound, so the goal is to read each weight from HBM as rarely as possible while producing as many useful tokens as possible per read.
5. Prefix/prompt caching — it reuses already-computed KV instead of recomputing prefill. It requires an exact shared prefix (and enough memory to keep that KV resident before eviction).