The Cost & Latency Dashboard

Lever 3 was the last pull. Now we lay all of it on one table — the discipline from the plan paying off, because every column changed by exactly one attributable thing. This is the artifact to keep: a dashboard you can stamp on any LLM feature to know what it costs and whether each change earned its place.

The full before/after

Stage	Input tok	Output tok	$/req (index)	Perceived TTFT	p99 e2e	Quality
Baseline (all-Opus, top-20, no cache, no stream)	~9,600	~400	1.00×	~4,600 ms	~7,000 ms	~92%
+ Lever 1: prompt caching	~1,860 billed	~400	~0.33×	~3,400 ms	~5,000 ms	~92%
+ Lever 2: model routing (70/30)	~1,860 billed	~400	~0.12×	~1,800 ms	~4,000 ms	~91%
+ Lever 3: trim + stream	~2,300	~300	~0.10×	~200 ms	~2,500 ms	~91%

Net result: roughly 10× cheaper per request, perceived TTFT roughly 20× faster, quality within a single point of baseline. No GPUs, no retraining, no infra migration — a cache header, a routing if, a reranker, and a streaming call.

Attribution: who saved what

Because we moved one lever at a time, each delta belongs to exactly one change:

$/req:  1.00× ──caching──▶ 0.33×  (≈3.0×, zero quality risk)
              ──routing──▶ 0.12×  (≈2.7× more, guarded by evals)
              ──trim─────▶ 0.10×  (≈1.2× more)
              (streaming: ~0× on $/req — it's a latency lever, not a cost one)

TTFT:   4,600 ms ──caching──▶ 3,400  (skip prefix prefill)
                 ──routing──▶ 1,800  (Haiku is faster on 70% of traffic)
                 ──trim─────▶ smaller prompt → faster prefill
                 ──stream───▶ ~200 ms  (the big one: show tokens immediately)

The lesson in the multiplication: these levers compound, not add. Each is a fraction, and fractions multiply — 0.33 × 0.36 × 0.83 ≈ 0.10. A single lever alone would have been a respectable win; stacked and measured, they’re an order of magnitude. Note too that streaming contributes nothing to the dollar column and everything to the perceived-latency column — which is exactly why we measured cost and TTFT in separate columns. A dashboard that collapsed them into one “performance” number would have hidden the most important UX win in the project.

What to monitor in prod

The dashboard is a snapshot; production drifts. Carry the same columns into live monitoring (this is the job of Part 8 — LLMOps):

• Cache hit rate — cache_read_input_tokens ÷ total prefix tokens. A silent invalidator (a stray timestamp, a reordered field) can drop it to zero and your bill quietly triples. Alert on it.
• Route distribution & escalation rate — the share going to each model. If “easy” traffic drifts toward Opus, the blended cost creeps back up.
• Token counts per request — input and output, p50 and p99. Context bloat creeps in as prompts evolve.
• TTFT and p99 end-to-end — the latency metrics that matter, as percentiles, because the tail is what users remember.
• Quality, continuously — keep running the eval set (and sample real traffic into it). Every cost lever can regress quality, and only the eval column catches it.

When to STOP optimizing

Optimization has diminishing returns, and past a point you’re trading real quality (or engineering time) for pennies. Stop when:

• You’ve hit the quality floor. The eval set is the wall. If the next trim, the next route, the next cap costs a quality point you can’t spare, you’re done — the floor is non-negotiable.
• Returns have gone diminishing. We went 1.00× → 0.33× → 0.12× → 0.10×. The first lever cut two-thirds of the bill; the last shaved only about a sixth off what remained (trimming falls under the cache floor, so it and caching partly cancel). When a lever’s payoff no longer justifies its complexity and its quality risk, ship and move on.
• The cost is no longer where the money is. Re-run the cost model. Once generation is cheap, your dollars may have moved to retrieval, embeddings, or egress — optimize that, not another 2% off a call that’s already an eighth of its original price.

Cheaper is a means, not the goal

The goal was never “minimize dollars” — it was “minimize dollars at fixed quality.” A 50× cost cut that drops the eval score five points is a failure, not a win. The quality column is the one number you don’t get to trade.

Where to go next

If app-layer levers have been pulled and cost still hurts, the next moves leave the application layer:

• Keep evaluating. The cheapest improvement is often a better eval set that lets you trim more safely — you can only optimize as aggressively as you can measure quality.
• Fine-tune a small model on your task so it answers the easy tier even cheaper than Haiku (see fine-tune vs RAG vs prompt) — worth it only at volume that amortizes the training cost.
• Self-host an open model when your traffic is steady and large enough that owning the GPU beats the per-token API bill — at which point you’re back in Parts 3–5, sizing KV caches and batches yourself. The API’s per-call meter becomes amortized hardware again.

The throughline, closed

We started with one question and never let go of it: what does this cost — in latency, memory, and dollars — and how do we make it cheaper? On this app the answers were concrete. Latency: TTFT fell ~20× — mostly from streaming, the lever that changed the experience without changing the work. Memory: the context window is the budget you own, and trimming it from ~9,600 to ~2,300 tokens cut both prefill latency and dollars at once. Dollars: ~10× cheaper per request, compounded from four measured levers, with quality held within a point.

That’s the whole discipline, portable to anything you build: name the costs, measure the baseline, pull one lever at a time, re-measure, and stop at the quality floor. The cheapest, fastest system is the one whose cost you actually understand.

Check your understanding

1. Why does the dashboard keep dollars-per-request and perceived TTFT in separate columns rather than one “performance” score?
2. Explain why the levers compound rather than add, using the cost-index numbers.
3. Which lever moved cost the least but mattered most, and to which metric?
4. Give three concrete signals you would monitor in production to detect that a lever has silently stopped working.
5. State two distinct conditions under which you should stop optimizing this app.

Show answers

1. Because some levers move only one of them — streaming cuts perceived TTFT but adds nothing to the dollar column. A single combined score would have hidden the project’s biggest UX win (and could let a cost regression masquerade as overall progress).
2. Each lever multiplies the remaining cost by a fraction (caching ×0.33, then routing ×~0.36 of that, then trimming ×~0.83 of that — trimming falls under the cache floor, so caching and trimming partly cancel), so they chain multiplicatively: 0.33 × 0.36 × 0.83 ≈ 0.10 — an order of magnitude, not the sum of three separate cuts.
3. Streaming — it contributed ~0× to dollars-per-request but delivered the largest latency win, collapsing perceived TTFT to roughly prefill time (~200 ms).
4. Cache hit rate (cache_read_input_tokens dropping toward zero), route/escalation distribution (easy traffic drifting onto the expensive model), and per-request token counts plus the continuous eval score (context bloat or a quality regression).
5. (a) You’ve hit the quality floor — the next change costs an eval point you can’t spare. (b) Returns have gone diminishing or the cost center has moved (re-run the cost model; the dollars may now be in retrieval/embeddings, not generation), so further app-layer tuning isn’t worth its complexity and risk.