Evals vs Efficiency

Observability showed you the cost side of every request. This page is about the side that does not show up on the latency graph: quality. The throughline of this book is making tokens cheaper — but almost every lever that makes a token cheaper can also make the answer worse. Evals are the instrument that measures the worse, so you can keep only the cheaper-without-worse wins.

Every efficiency lever is a quality bet

Walk back through the book and notice that each optimization quietly trades against accuracy:

Efficiency lever	What it saves	What it can cost
Quantization (FP16 → INT4)	memory, dollars	precision on hard prompts, reasoning
Routing to a smaller model	dollars, latency	accuracy on the long tail
Fewer RAG chunks	prefill tokens, TTFT	grounding, recall of the right fact
Prompt compression	input tokens	dropped instructions, lost nuance
Aggressive prompt caching	prefill cost	stale context if invalidation is wrong
Speculative decoding	latency	nothing if implemented right (it is exact)

That last row matters: a few optimizations are quality-neutral by construction (speculative decoding produces identical tokens; PagedAttention just packs memory). But most are not. The discipline is to know which kind each one is — and to measure the ones that are bets rather than assume they are free.

The core rule of this part

Never cut cost without measuring the quality delta. “We quantized to INT4 and saved 40% on GPUs” is half a sentence. The whole sentence is “…and accuracy on our eval set went from 87% to 85%, which we decided was worth it.” Without the second clause you have not made a decision — you have made a guess and called it a win.

Build a representative eval set

You cannot measure quality against vibes. An eval set is a fixed collection of inputs with a way to score the outputs — your regression test for intelligence. What makes it trustworthy:

• Representative. Sample from real production traffic (your logs), including the hard tail, not just easy happy-path cases. An eval set of softballs greenlights changes that break in production.
• Stable and versioned. The set must not drift, or you cannot compare today’s score to last month’s. Version it like code.
• Scored by something. Exact-match or unit tests where possible (code, JSON schema, math answers); rubric-based judgment where not (helpfulness, tone, faithfulness).
• Sized for signal. Big enough that a 1-2 point move is not noise. A few hundred examples is a reasonable floor; a handful is theater.

Offline evals, then online

Two stages, because each catches what the other misses.

Offline — run the candidate (quantized model, new prompt, fewer chunks) against the frozen eval set before shipping. Cheap, fast, deterministic, reproducible. It is your CI gate: if the score drops below threshold, the change does not ship.

Online — offline sets never fully match live traffic, so confirm on real users at low risk:

• Canary: route a small slice (say 5%) of production traffic to the new config and watch quality and cost metrics. Roll back automatically if quality alarms fire.
• A/B test: run old and new side by side and compare real outcomes — thumbs-up rate, task completion, retry rate, escalation to a human. These business metrics are the ground truth that an eval score only approximates.

efficiency change ──► OFFLINE eval (frozen set)
                          │ pass?
                          ▼
                      CANARY 5%  ──► metrics + quality OK?
                          │ yes
                          ▼
                       A/B 50/50 ──► real-outcome win?
                          │ yes
                          ▼
                       full rollout
   any stage fails → roll back, keep the old (more expensive) config

LLM-as-judge, and its caveats

When there is no exact answer to match, a common trick is to have a strong model score the candidate’s output against a rubric — “rate faithfulness to the source 1-5.” It scales cheaply and correlates with human judgment well enough to be useful. But it is a measurement instrument with known biases, and treating its score as ground truth will burn you:

• Position/verbosity bias — judges favor the first option shown, and longer, more confident answers, regardless of correctness.
• Self-preference — a model tends to rate its own family’s outputs higher.
• It costs tokens too — judging is itself inference; a big eval run on a frontier judge has a real bill, so the measurement has a cost-versus-fidelity trade of its own (cheaper judge vs more reliable judge).
• Calibrate it. Spot-check the judge against human labels on a sample; if it disagrees with humans, fix the rubric before trusting the score.

Cost lens on evals

Evals are not free — an eval set is inference you pay for, and an LLM judge doubles that. But the spend is trivially justified: it is the insurance premium on every efficiency change. The alternative — shipping a 40% cost cut that silently dropped accuracy 10% and churned your best customers — is far more expensive. Budget evals as a fixed cost of doing optimization at all.

Regression-test efficiency changes

Fold the eval into the deployment pipeline so quality cannot regress unnoticed. Treat the eval score exactly like a test suite: every change to model, quantization, prompt, retrieval depth, or router gets run through the frozen set, and a score drop past a threshold blocks the merge the same way a failing unit test would. This is how you make the core rule automatic instead of aspirational — the cheaper config only ships if the quality held.

State the trade explicitly, every time: cost saved vs quality lost. Sometimes 2 points of accuracy for 40% off the bill is an obvious yes; sometimes 2 points is the difference between a usable product and a broken one. Evals do not make the decision for you — they make sure you are deciding with numbers on both axes instead of one.

It actually happened — quantifying the judge’s biases (2023)

The biases above are not folklore; they were measured. In Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), researchers tested using GPT-4 to grade other models’ answers and found its verdicts agreed with human preferences at roughly the rate two humans agree with each other — strong enough to be useful at scale. But the same study isolated the failure modes this page warns about: position bias (the judge favored whichever answer was shown first, and swapping the order could flip the verdict), verbosity bias (longer answers scored higher even when not better), and self-enhancement bias (a model nudged its own family’s outputs up). The lesson that stuck: an LLM judge is a useful instrument with a calibration sheet, not an oracle — swap answer positions, control for length, and spot-check against humans before you trust the score.

With quality guarded, the last question is the biggest dollar lever of all: should you run the model yourself or rent it? That is cost modeling.

Check your understanding

1. Why does almost every efficiency lever in this book need to be guarded by an eval? Give two examples of levers and the quality they risk.
2. Which optimizations are quality-neutral by construction, and why does that distinction matter?
3. What three properties make an eval set trustworthy?
4. Why run both offline evals and online canaries/A-B tests rather than just one?
5. Name two biases of LLM-as-judge and one way to keep it honest.

Show answers

1. Each lever trades accuracy for cost/latency, and the quality loss is invisible on the cost dashboard — so you must measure it directly. Examples: quantization risks precision on hard reasoning; fewer RAG chunks risks grounding/recall; a smaller routed model risks long-tail accuracy; prompt compression risks dropped instructions.
2. Speculative decoding (exact same tokens) and PagedAttention (just memory packing) do not change outputs. The distinction matters because you should spend eval effort on the genuine bets, not on neutral changes you can ship freely.
3. Representative of real traffic including the hard tail; stable and versioned so scores are comparable over time; large enough that a small move is signal not noise (with a real scoring method).
4. Offline is cheap, fast, and deterministic but never fully matches live traffic; online (canary/A-B) confirms on real users with real outcome metrics at low risk. Each catches failures the other misses.
5. Biases: position/order bias, verbosity/length bias, self-preference (favoring its own family). Keep it honest by calibrating against human labels on a sample and fixing the rubric when the judge disagrees with humans.