Revision · The Capstone

The capstone stops explaining levers and starts pulling them: one real document-Q&A RAG app on the Claude API, a meter on every token, and a bill cut one measured step at a time. The goal is precise — cut dollars-per-request and time-to-first-token without sacrificing answer quality — and the discipline is measuring before and after every change.

What this part covered

• The plan: build, measure, optimize — baseline first, then one lever at a time, re-measuring after each, so every win is attributable to a single change rather than superstition.
• The metrics that define the app — tokens in/out, dollars per request, time-to-first-token, end-to-end latency, and a quality score from a small eval set; on a hosted API your “memory” budget is the context window, not HBM.
• The naive baseline — retrieve a lot, stuff one big prompt, call one strong model, no caching, no streaming; it works on day one and is quietly expensive forever, which is exactly where the waste hides.
• Lever 1 — prompt caching — the system prompt and session context repeat on every call, so caching that stable prefix bills the repeated input tokens at a fraction and skips their prefill, cutting both cost and TTFT.
• Lever 2 — model routing & cascades — most questions are easy, so send those to a cheap model (Haiku) and reserve the strong one (Opus) for the hard minority; the blended cost drops, guarded by the eval set so quality holds.
• Lever 3 — context trimming & streaming — send the top few chunks instead of twenty and cap the output (fewer tokens in and out, and RAG showed extra context can even hurt quality), then stream so the user sees the answer immediately.
• The cost & latency dashboard — the full before/after, compounded and attributed lever by lever, plus what to monitor in prod, when to stop optimizing, and where to go next.

The takeaway

These are the cheapest efficiency wins you will ever ship: a cache header, a routing if-statement, a reranker, and a streaming call — no GPUs, no retraining, no infra migration. The lasting deliverable is a mental dashboard for any LLM feature: the four or five numbers that define its cost and latency, the app-layer levers that move them, and the discipline to measure before and after each change. That closes the book’s throughline where it lives for most builders — the application layer, where the bill is recurring and paid on call one million.