The Plan: Build, Measure, Optimize

The overview named the goal — cut dollars-per-request and TTFT without losing quality. This page fixes the method that gets us there honestly: what exactly we build, which numbers we track, and the strict order we pull the levers so that each saving can be pinned to the lever that caused it.

The discipline: measure before you optimize

The single rule that makes this capstone trustworthy: never change two things at once, and always re-measure. If you cache and route and trim in one commit and the bill drops, you’ve learned nothing about which lever did the work — and you can’t tell that one of them quietly hurt quality. So we move in lockstep:

build baseline ─▶ MEASURE ─┐
                           ▼
        apply Lever 1 ─▶ MEASURE ─┐   (attribute the delta to Lever 1)
                                  ▼
              apply Lever 2 ─▶ MEASURE ─┐   (attribute the delta to Lever 2)
                                       ▼
                    apply Lever 3 ─▶ MEASURE   (attribute the delta to Lever 3)

This is the same instinct as load testing before tuning a server: a number you didn’t generate honestly is worse than no number, because it gives false confidence.

The app

A doc-Q&A RAG assistant, in two halves:

• Index time (once): chunk the documents, embed each chunk with an embedding model, and store the vectors. Embeddings are cheap and computed once, then reused on every query — exactly the “spend a little on retrieval” trade from RAG efficiency.
• Query time (every request): embed the question, retrieve the most relevant chunks, build a prompt, and call Claude’s Messages API for the answer.

┌─ index time (once) ─────────────┐   ┌─ query time (every request) ──────────────┐
│ docs → chunks → embed → store    │   │ question → embed → retrieve → prompt → Claude │
└──────────────────────────────────┘   └────────────────────────────────────────────┘

The Claude call uses the official anthropic SDK: client.messages.create(model=..., max_tokens=..., system=[...], messages=[...]). That one call is where the meter runs, so it is where every lever aims.

The metrics we track

After each step we record the same row. These are the numbers that define the app’s cost and latency — the application-layer counterpart to TTFT, TPOT, and goodput:

Metric	What it measures	Why we track it
Input tokens / request	prompt size billed per call	the recurring dollars knob; also prefill latency
Output tokens / request	generated tokens billed per call	output costs several× input — the priciest tokens you buy
Dollars / request	blended cost per answer	the headline cost number
TTFT (perceived)	time until the user sees content	the headline UX number
End-to-end p99	the unlucky-1% total latency	tails are what users remember
Quality (eval pass %)	answers graded against a small eval set	the guardrail — none of the above may be bought with this

The eval set is the quietest but most important column: a few dozen questions with known-good answers, scored automatically (exact match, an LLM-judge, or a rubric). Without it, “cheaper” and “worse” are indistinguishable.

The quality floor is non-negotiable

Every lever below can, pushed too hard, trade quality for cost — a stale cache, a mis-route, an over-trimmed context. The eval set is the floor that catches it. We tighten each lever only until quality dips, then stop.

The levers, in order

We pull them cheapest-and-safest first:

1. Prompt caching — the system prompt and stable context repeat across a session’s turns and across users. Cache that prefix and the repeated input tokens bill at a small fraction, and the prefill is skipped. Pure win, near-zero risk.
2. Model routing — most questions are easy. Send those to a cheap model (claude-haiku-4-5) and reserve the expensive one (claude-opus-4-8) for the hard minority. Big win, guarded by the eval set against mis-routing.
3. Context trimming + streaming — rerank and send the top 3 chunks instead of 20, summarize history, cap max_tokens (fewer tokens in and out); then stream the answer so the user sees it immediately (perceived latency collapses even when total time barely moves).

Tech sketch

embedding model + vector store   →  retrieval
        anthropic SDK            →  generation (Messages API; .stream() for streaming)
        usage.* fields           →  the meter (input/output/cache tokens per call)
        small eval harness       →  the quality guardrail

Every cost number we report comes straight from the API response’s usage object — input_tokens, output_tokens, and the cache fields — multiplied by current per-token prices (always check current pricing; we keep the math in relative units so the shape of the win is what you remember).

With the method fixed, we build the thing we’re about to optimize: the naive baseline.

Check your understanding

1. Why does the method forbid changing two levers in the same step?
2. Name the two halves of the RAG app and say which costs are paid once versus on every request.
3. Which tracked metric is the guardrail, and what failure does it catch that the cost metrics cannot?
4. In what order are the three levers applied, and what’s the rationale for that order?
5. Where do the reported cost numbers actually come from in the API response?

Show answers

1. Because if several changes ship together you can’t attribute the resulting delta to any one lever, and you can’t tell whether one of them quietly degraded quality. One change per step keeps every win (and every regression) traceable.
2. Index time (chunk → embed → store) is paid once and reused; query time (embed question → retrieve → prompt → Claude) is paid on every request. The per-request half is where optimization concentrates.
3. The eval pass-rate (quality). It catches a “cheaper” change that is actually a “worse” change — a stale cache, a mis-route, or over-trimmed context — which the dollar and latency columns would happily report as a win.
4. Prompt caching, then routing, then context trimming + streaming — cheapest-and-safest first (caching is near-zero risk), escalating to levers that need an eval guardrail (routing, trimming).
5. The response’s usage object: input_tokens, output_tokens, and the cache fields (cache_read_input_tokens, cache_creation_input_tokens), multiplied by current per-token prices.