Revision · Part 3 · Inference Efficiency Core

Part 3 is the heart of the book: the handful of techniques that took the same hardware and the same model and improved real-world throughput and latency by an order of magnitude. Every one is a mechanical response to one fact — decode is memory-bound, so the game is to read each weight from HBM as few times as possible and serve as many tokens as possible per read.

What this part covered

• KV cache math sizes the dominant cost — 2 × n_layers × n_kv_heads × head_dim × seq_len × batch × bytes grows linearly in context and batch (~0.78 MB/token for Llama-2-13B), and it — not the weights — caps how many requests a GPU holds (~35 on an 80 GB A100).
• GQA/MQA shrink the cache at the source — cutting n_kv_heads (MQA to 1, GQA to a small group) shrinks KV proportionally while keeping full query heads for quality; a 5× cut buys ~5× concurrency, baked in at training time.
• Continuous batching keeps lanes full — static batching holds every request hostage to the longest in its batch (idle lanes + padding waste); iteration-level scheduling evicts finished requests and admits waiting ones every token step, lifting utilization ~50% → ~90% (≈1.8× more tokens/sec).
• The throughput-versus-latency trade — a larger batch amortizes the weight-read across more tokens (lower $/token) but raises each user’s TPOT, so the operating point is chosen on purpose per product.
• PagedAttention eliminates KV waste — treating the cache like OS virtual memory (fixed-size blocks, allocated on demand, addressed via a per-request block table) replaces contiguous max-length reservation, cutting waste from 60–80% to ≤15 tokens per request and buying ~8× concurrency; copy-on-write lets requests share identical blocks.
• FlashAttention moves fewer bytes, not fewer FLOPs — tiling Q/K/V through on-chip SRAM with an online softmax fuses the passes so the n×n score matrix never touches HBM, cutting attention memory from O(n²) to O(n) and running 2–4× faster — losslessly, the identical math.
• Speculative decoding buys latency from idle compute — a small draft model proposes k tokens, the big model verifies them all in one prefill-like pass; at α=0.8, k=4 that yields ~3.36 tokens per target step (2–3× lower latency), lossless via rejection sampling, best at low batch.
• Prefix/prompt caching skips work entirely — because transformers are causal, a shared prefix’s KV is bit-for-bit identical across requests, so caching it once cuts TTFT ~40× and skips ~52 PFLOP of redundant prefill across 1,000 requests; the same mechanism the Claude API exposes as prompt caching.
• These stack, they aren’t alternatives — production engines like vLLM run PagedAttention, continuous batching, FlashAttention kernels, and automatic prefix caching together, with speculative decoding on top, and the gains multiply — the reason per-token prices have fallen so fast.

The takeaway

Six angles on one idea: batching amortizes the weight-read across requests, paging lets you batch more, FlashAttention cuts the reads attention needs, speculative decoding gets multiple tokens per read, and prefix caching skips reads entirely. Together they turn a notebook demo into a system that serves millions of requests at a price someone will pay. From here the book moves outward — to model-level compression and the application layer, where the cheapest token is the one you never generate.