Revision · Measuring & Operating (LLMOps)

Part 8 is where earlier parts’ levers get audited: LLMOps is the discipline of measuring a running system before you touch it, then watching what your change does to the numbers that matter. Its iron rule — establish a baseline, change one thing, re-measure, keep the win only if quality held.

What this part covered

• Measure before you optimize — optimization without a baseline is gambling; a “20% latency win” is meaningless until you say which latency, at what percentile, and under what load, and every efficiency lever can silently cost quality that never shows on the latency graph.
• The metrics that matter — TTFT (prefill-dominated, the blank-screen wait) and TPOT (decode-dominated, the reading speed) are separate numbers with separate fixes; for any real answer length, TPOT × output tokens is the bulk of wall-clock time.
• Goodput, not raw throughput — the metric that pays the bills counts only requests meeting their SLO; cramming the batch can raise raw RPS while delivering fewer useful requests, and you always report percentiles (p50/p95/p99), never a mean that launders the tail.
• Load-test honestly — input/output length distribution is the experiment (it sets the prefill-vs-decode mix), so sample real shapes, sweep concurrency to find the knee (the SLO-respecting operating point), discard warmup, and beware coordinated omission from closed-loop generators.
• Observability instruments the token — classic CPU/memory/200-OK metrics are blind to the unit of cost; watch tokens in/out, cost-per-request, KV-cache utilization (the real admission gauge), queue depth, cache hit rate, and batch size, and trace each request through retrieve → prefill → decode as separate spans.
• Attribute the spend — tag requests by customer, feature, and model so total spend becomes actionable; low GPU utilization is not “fine,” it is money on fire.
• Evals guard quality — every cost cut is a bet against accuracy (except quality-neutral levers like speculative decoding and PagedAttention); run a representative, versioned, well-sized set offline as a CI gate, then confirm online with canaries and A/B tests, and treat LLM-as-judge as a biased instrument to calibrate, not an oracle.
• Cost modeling: build vs buy — API is pure variable cost (line through origin, steep slope), self-host is a fixed GPU floor with a shallow per-token slope; they cross at a break-even volume, and utilization decides everything — the same GPU and code can be 5× cheaper or 5× dearer per token.

The takeaway

Part 8 turns the book’s throughline from a rhetorical “what does this cost?” into a measurable “how would I know if my change actually helped?” — a metric, a load test, a trace, an eval, and a break-even spreadsheet. You can now instrument a serving stack, load-test it to its SLO-respecting capacity, watch it in production, guard its quality, and decide with a dollar figure whether to run it yourself or rent it — the exact bridge into the capstone.