Revision · Serving Systems

Part 5 stopped deriving techniques and started assembling them: it wires the Part 3 inference tricks and Part 4 compression into one always-on service that takes concurrent requests over an API and answers each inside a latency budget. The single fact driving every trade-off — the model and its KV cache live in small, expensive, slow-to-fill HBM — shows up in the scheduler, the parallelism layout, and autoscaling alike.

What this part covered

• The serving stack — off-the-shelf engines (vLLM, TGI, TensorRT-LLM, SGLang) bundle paged KV, continuous batching, and quantization behind an OpenAI-compatible API; the choice is which extra trick and operational style fits your traffic, and matching engine to workload can swing dollar-per-token ~40% on the same GPU.
• Request scheduling and queueing — requests arrive at random, so a scheduler spends two budgets (compute per step, KV memory in HBM) against a queue, deciding who runs each GPU iteration since Orca made scheduling per-token in 2022.
• Admission and preemption — a request joins only if KV space is free; when HBM fills the scheduler preempts a victim, recovering its KV by recompute (costs FLOPs) or CPU swap (costs PCIe bandwidth).
• Prefill vs decode contention — compute-heavy prefill and memory-bound decode fight for the GPU; favoring one spikes the other’s latency (TTFT vs TPOT), and chunked prefill interleaves them so neither monopolizes.
• Throughput vs latency vs SLO — a latency SLO is a cap on batch size in disguise; queue depth is the load signal, and SLO-aware scheduling trades raw tokens/sec for bounded tail latency.
• Tensor, pipeline, and data parallelism — TP splits each weight matrix (frequent all-reduce → NVLink, intra-node), PP splits layers (rare handoff → crosses nodes, pays the pipeline bubble shrunk by micro-batches), DP replicates for throughput; the same all-reduce is ~16× costlier off NVLink.
• Multi-GPU and multi-node serving — the frontier topology is TP inside each node, PP across nodes, with the KV cache sharded across ranks; placement is first-class because a mis-placed TP group turns a 0.4 ms all-reduce into ~6.4 ms — a 405B FP16 model spans two nodes, but INT8 pulls it back into one.
• Autoscaling and cold starts — replicas should track bursty traffic, but loading a 140 GB model into HBM takes 28 s–3 min, so you hide the latency (warm pools, min-replicas, NVMe caching) or shrink the bytes (quantize), and scale on the symptom (queue depth, p95 TTFT), not GPU utilization.

The takeaway

By the end you can look at a model size, a traffic pattern, and a latency SLO and sketch a plan: which engine, how many GPUs in what parallel layout, how to schedule the queue, and how many replicas to keep warm — each with a rough dollar figure attached. The recurring lesson is that the same model served two ways can differ several-fold in dollar-per-token purely from how it was scheduled and placed, because HBM scarcity is the constraint underneath all of it.