Part 9 · Advanced & Rare Concepts

Everything so far has built a working mental model: the two phases, the KV cache, batching and scheduling, compression, and why inference is memory-bound. Part 9 is the senior track: the techniques and failure modes that only show up when you are serving real traffic at scale, on real hardware, for real money. The throughline does not change — what does this cost in latency, memory, and dollars, and how do we make it cheaper? — but the answers get sharper and stranger.

What “advanced” means here

These are not exotic for their own sake. Every page in this part exists because a naive serving stack hits a wall that the earlier tricks alone cannot break:

• A long prompt arrives and freezes everyone else’s token stream.
• Context windows grow to 128K+ and the KV cache eats more memory than the weights.
• You need to serve 200 fine-tuned variants and cannot afford 200 copies of a 70B model.
• One GPU is forced to do two jobs with opposite bottlenecks at the same time.
• The model has to run on a phone with no datacenter GPU in sight.
• The GPU itself turns out to be the wrong chip for memory-bound decode.

Each is a scaling failure mode first, and a clever fix second.

The pattern behind the pages

Most of Part 9 is one idea applied in different places: stop forcing one resource to do incompatible jobs, and stop paying for exactness you don’t need.

  Separate the jobs            Approximate what you can afford to
  ────────────────────        ──────────────────────────────────
  prefill ≠ decode      →      sliding-window / sink attention
  base ≠ adapter        →      KV eviction (keep important tokens)
  GPU pool A ≠ pool B   →      INT4 on-device
                              specialized silicon for one phase

Roadmap

1. Disaggregated prefill & decode — run the compute-bound and memory-bound phases on separate GPU pools so a prefill spike stops hurting everyone’s token stream; pay only the KV-transfer cost.
2. Long-context efficiency — attention is O(n²) and the KV cache is O(n); sliding windows, attention sinks (StreamingLLM), ring attention, KV eviction (H2O), and GQA/MQA buy back the memory and compute.
3. Multi-LoRA serving — host hundreds of fine-tuned “models” for roughly the memory of one base by serving LoRA adapters over a shared backbone (S-LoRA, Punica).
4. Chunked prefill — slice a giant prefill into pieces interleaved with ongoing decode, smoothing tail latency and filling idle compute.
5. On-device & edge inference — privacy, offline, and zero per-token cost on a phone or laptop, paid for with heavy INT4 quantization and small models.
6. Beyond GPUs — TPUs, LPUs (Groq), wafer-scale (Cerebras), and AWS Inferentia/Trainium, and why specialized silicon can beat a GPU at the memory-bound decode problem.

The mental model

Parts 3–4 made each request cheaper. Part 9 makes the fleet cheaper and more predictable — by separating workloads that fight each other, and by refusing to pay full price for full precision and full attention when an approximation is good enough.

Note

Many of these techniques are young and move fast. Treat specific system names (Mooncake, S-LoRA, Groq) as representative examples of an idea, not as the final word. The principle — match the resource to the job — outlives any one implementation.

By the end you should be able to look at a latency graph, spot which phase or resource is being abused, and name the lever that un-abuses it without quietly tripling your bill.

Check your understanding

1. What single recurring idea connects disaggregation, multi-LoRA serving, and separate GPU pools?
2. Why does long context become a memory problem and not just a compute problem?
3. Give one concrete scaling failure mode that motivates this part.
4. What is the throughline question this whole textbook keeps asking?
5. Why are the specific system names (e.g. Mooncake, S-LoRA) described as examples rather than fixed answers?

Show answers

1. Stop forcing one resource to do two incompatible jobs — separate prefill from decode, base from adapter, one GPU pool from another — so each can be tuned for its own bottleneck.
2. The KV cache grows O(n) with context length and must stay resident in memory for every token of decode, so at 128K+ tokens it can exceed the size of the model weights themselves.
3. Examples: a long prompt’s prefill freezing everyone else’s token stream; the KV cache outgrowing VRAM at long context; needing 200 fine-tuned variants without paying for 200 full model copies.
4. What does this cost — in latency, memory, and dollars — and how do we make it cheaper?
5. The field moves quickly; the durable lesson is the principle (match the resource to the job), while any specific implementation may be superseded.