Part 5 · Serving Systems

Parts 3 and 4 handed you a kit of parts. Part 3 gave the inference techniques — KV-cache sizing, continuous batching, PagedAttention, FlashAttention, speculative decoding, prefix caching. Part 4 gave compression — quantization that shrinks the weights so they fit on fewer, cheaper GPUs. Each is a lab demo on its own. A production system is what you get when you wire them into one always-on server that takes HTTP requests from thousands of strangers and answers every one inside a latency budget, on hardware you are renting by the second. That wiring is Part 5.

What a serving system is for

A notebook that calls model.generate() serves one request at a time and crashes if two arrive together. A serving system turns that model into a service: it loads the weights once, keeps them resident in HBM, accepts concurrent requests over an OpenAI-compatible API, packs them into batches, schedules every GPU iteration, and meters the whole thing so you can see what it costs. Everything in this book has been building toward making that service cheap — and the cost throughline gets its most concrete answers here, because this is where latency, memory, and dollars are all measured at once on real traffic.

The single hardest fact this part keeps colliding with: the model and its KV cache live in GPU HBM, which is small, expensive, and slow to fill. That one fact drives the scheduler (it fights over KV space), the parallelism strategy (the model may not fit on one card), and autoscaling (you cannot spin up a GPU as fast as a web server). Hold it in mind as the source of nearly every trade-off ahead.

The roadmap

Read these in order; each adds a layer to the server.

1. The Serving Stack — the off-the-shelf engines (vLLM, TGI, TensorRT-LLM, SGLang) that already implement Part 3 and Part 4 for you. What each one optimizes, and which to reach for.
2. Request Scheduling & Queueing — requests arrive at random; something must decide who runs each GPU iteration. Admission, preemption, fairness vs throughput, and how prefill and decode fight over the same silicon.
3. Tensor & Pipeline Parallelism — what to do when one model is too big or too slow for a single GPU. Splitting weight matrices (tensor) vs splitting layers (pipeline) vs replicating (data), and the communication tax each one pays.
4. Multi-GPU & Multi-Node Serving — combining those strategies across NVLink inside a node and the network between nodes, where placement decides whether you pay the slow-wire tax on every token.
5. Autoscaling & Cold Starts — traffic is bursty and GPUs are expensive, so you want to scale replicas to load. But loading a 140 GB model into HBM takes tens of seconds, so you cannot scale instantly. The mitigations and their cost-versus-latency trade.

This part assembles, it does not re-derive

We will not re-prove PagedAttention or quantization — Parts 3 and 4 did that. Here those results are components. The new material is the systems engineering around them: queues, schedulers, collective operations, replicas, and the network.

The question to keep asking

On every page: what does this cost in latency, memory, and dollars, and how do we make it cheaper? In Part 5 the answers stop being per-technique and become per-system — the same model served two different ways can differ several-fold in dollar-per-token, entirely because of how it was scheduled and placed.

By the end you will be able to look at a model size, a traffic pattern, and a latency SLO, and sketch a serving plan: which engine, how many GPUs in what parallel layout, how to schedule the queue, and how many replicas to keep warm — with a rough dollar figure attached to each choice.

Check your understanding

1. What does a serving system add on top of a notebook’s model.generate() call?
2. Which single hardware fact drives most of the trade-offs in this part?
3. Name the five layers the roadmap stacks up, in order.
4. Why does this part not re-derive PagedAttention or quantization?
5. How does the cost throughline change between Part 3 and Part 5?

Show answers

1. It loads the weights once and keeps them resident, accepts many concurrent requests over a (typically OpenAI-compatible) API, batches and schedules them across GPU iterations, manages the KV cache, and meters cost — turning a one-shot model call into an always-on service.
2. The model and its KV cache live in small, expensive, slow-to-fill GPU HBM; that constraint drives the scheduler, the parallelism strategy, and autoscaling.
3. Serving stacks → scheduling & queueing → tensor/pipeline parallelism → multi-GPU/multi-node → autoscaling & cold starts.
4. Because Parts 3 and 4 already proved them; here they are reused as components, and the new content is the systems engineering (queues, schedulers, collectives, replicas, network) that assembles them.
5. In Part 3 cost was measured per technique on one GPU; in Part 5 it is measured per system on real traffic, where scheduling and placement alone can swing dollar-per-token several-fold.