Part 8 · Measuring & Operating (LLMOps)

Parts 3 through 7 handed you levers: continuous batching, quantization, PagedAttention, parallelism, prompt and context economy. Every one of them promised to make a token cheaper. Part 8 is where you find out whether it actually did — and at what cost to latency and quality. This is LLMOps: the discipline of measuring a running system before you touch it, and watching what your change does to the numbers that matter.

Measure before you optimize

The fastest way to waste a week is to optimize a system you cannot see. You quantize a model to save memory, ship it, and three days later support tickets pile up — was it slower, or just wrong? You will not know, because you never measured the baseline. The iron rule of this part: establish the numbers first, change one thing, re-measure, keep the win only if the quality held. Optimization without measurement is gambling with extra steps.

The cost throughline of this whole book — what does this cost in latency, memory, and dollars, and how do we make it cheaper? — only becomes answerable once you can read it off a dashboard. A “20% latency win” is meaningless until you can say which latency (first token or per-token?), at what percentile (the median or the unlucky 1%?), and at what load (one user or five hundred?). Part 8 makes those questions precise.

The trap this part exists to prevent

Every efficiency lever can cost quality, and the quality loss is usually invisible on the latency graph. A smaller model is faster and dumber. Fewer retrieved chunks is cheaper and less grounded. You must measure both sides of every trade — never just the side that looks good.

The roadmap

Read these in order; each layer assumes the one before it.

1. The Metrics That Matter — TTFT, TPOT, end-to-end latency, throughput, and the one metric that actually matters: goodput, the requests that meet their SLO. Percentiles over averages, and the latency-versus-throughput trade restated as a measurement problem.
2. Load Testing & Benchmarking — how to benchmark honestly: realistic input/output shapes, concurrency sweeps that find the knee, warmup and steady state, and the pitfalls that make vendor numbers lies.
3. Observability for AI Systems — what to monitor beyond CPU and memory: tokens in/out, cost per request, cache hit rate, queue depth, KV-cache utilization, and tracing a request through retrieve → prefill → decode.
4. Evals vs Efficiency — the guardrail. Every cost cut is a hypothesis about quality; evals are how you test it. Offline sets, online canaries, LLM-as-judge and its caveats, and regression-testing efficiency changes.
5. Cost Modeling: Build vs Buy — the unit economics. API dollars-per-token versus self-hosted dollars-per-GPU-hour, the break-even volume between them, and why utilization decides everything.

The question to keep asking

On every page: what does this cost, and how would I know if my change actually helped? In Part 8 the second clause stops being rhetorical — it becomes a metric, a load test, a trace, an eval, and a spreadsheet.

By the end you will be able to take a serving stack from Part 5, instrument it, load-test it to find its SLO-respecting capacity, watch it in production, guard its quality with evals, and decide — with a dollar figure — whether to run it yourself or rent it from an API. That last decision is the bridge into the capstone.

Check your understanding

1. What is the “iron rule” this part opens with, and why does optimization without measurement fail?
2. Why is a bare claim like “20% latency win” not yet meaningful?
3. What is the trap that makes Evals vs Efficiency a necessary page rather than an optional one?
4. List the five stages of the roadmap in order.
5. How does the cost throughline change character in Part 8 compared to earlier parts?

Show answers

1. Establish a baseline, change one thing, re-measure, and keep the win only if quality held. Without a baseline you cannot attribute any change — you are gambling, not engineering.
2. Latency is ambiguous: you must specify which latency (TTFT vs TPOT), at what percentile (p50 vs p99), and under what load (one user vs hundreds) before the number means anything.
3. Every efficiency lever can silently cost quality, and that loss does not show up on the latency graph — so you must measure quality alongside cost on every change.
4. Metrics → load testing → observability → evals → cost modeling.
5. Earlier parts asked “what does this cost?”; Part 8 makes the answer measurable and adds the second clause — “how would I know if my change actually helped?” — turning it into dashboards, load tests, traces, evals, and break-even math.