Revision · Advanced & Rare Concepts

Part 9 is the senior track: techniques and failure modes that only surface when you serve real traffic at scale, on real hardware, for real money. The throughline never changed — what does this cost in latency, memory, and dollars, and how do we make it cheaper? — but the recurring move is one idea: stop forcing one resource to do two incompatible jobs, and stop paying for exactness you can’t afford.

What this part covered

• Disaggregated prefill & decode — the compute-bound prefill phase and memory-bound decode phase interfere on one GPU, so a prefill spike freezes everyone’s token stream; splitting them onto separate pools fixes it, and you pay only the KV-transfer cost.
• Long-context efficiency — attention is O(n²) in compute and the KV cache is O(n) in memory, so 128K+ contexts get expensive on both axes; sliding windows and attention sinks (StreamingLLM) cut compute, KV eviction (H2O) and GQA/MQA cut memory, and ring attention spreads the sequence across devices.
• Multi-LoRA serving — hosting N fine-tuned variants as N full model copies is unaffordable; serving lightweight LoRA adapters over one shared base (S-LoRA, Punica) gives you hundreds of “models” for roughly the memory of one, with batching across adapters.
• Chunked prefill — one giant prefill blocks every other request’s decode; chopping it into chunks interleaved with ongoing decode smooths tail latency and, as a bonus, uses the idle compute that memory-bound decode leaves on the table.
• On-device & edge inference — running locally buys privacy, offline use, and zero per-token cost, paid for with heavy INT4 quantization and small models against tight memory and compute limits.
• Beyond GPUs — TPUs (systolic arrays), LPUs (Groq’s deterministic in-SRAM decode), wafer-scale silicon (Cerebras), and cloud ASICs (Inferentia/Trainium) can beat a GPU at memory-bound decode — but the real cost is the ecosystem, not just the FLOPs.

The takeaway

Every page here is the same principle applied to a different resource: match the resource to the job. Separate prefill from decode, base from adapter, one GPU pool from another; approximate attention and precision when a full-price version buys you nothing. Treat specific system names as representative examples, not final answers — the durable skill is reading a latency graph, spotting the abused phase or resource, and naming the lever that un-abuses it without tripling the bill. Next, Part 11 moves this same question to the frontier, where the cost driver stops being one call.