Overview — The Capstone

Nine parts of this book asked one question over and over: what does this cost — in latency, memory, and dollars — and how do we make it cheaper? We asked it of GPUs, of the KV cache, of batching and quantization, and then — in Part 6 — of the application layer, where you call a model API instead of owning the GPU. This capstone stops explaining levers and starts pulling them. We build one real application on the Claude API, put a meter on it, and cut its bill in front of you, one measured step at a time.

What we’re building

A document-Q&A assistant: a Retrieval-Augmented Generation app that answers questions about a corpus of documents. It is the most common shape of real LLM product — a support bot over a knowledge base, an internal “ask the handbook” tool, a research assistant over your own PDFs — so the lessons transfer directly to whatever you ship.

user question
     │
     ▼
 embed + retrieve relevant chunks ──▶ build a prompt ──▶ Claude (Messages API) ──▶ answer
     │                                                                              │
     └──────────────── a meter runs on every token in and every token out ─────────┘

The whole point is that meter. On a hosted API you can’t touch the kernels or the scheduler, but every choice you make about what to send, how often, to which model, and in what shape changes the bill — and that bill is recurring, paid on call one million, not just call one.

The goal, stated as a number

The naive version of this app works on day one and is quietly expensive forever. Our goal is precise:

Cut dollars-per-request and time-to-first-token without sacrificing answer quality.

All three clauses matter. Cutting cost by sending a smaller, dumber model is easy — and useless if the answers get worse. The discipline of this capstone is holding quality fixed (we measure it with a small evaluation set) while the cost and latency fall.

The method, in one line

Baseline first, then one lever at a time, re-measuring after each. We refuse to optimize anything until we’ve measured it, because un-measured optimization is just superstition — you can’t claim a win you never weighed. So the capstone has a strict shape:

1. Build the naive RAG app and measure it: tokens in/out, dollars per request, TTFT, end-to-end latency, and a quality score.
2. Apply Lever 1 — prompt caching: stop re-billing the stable prefix. Re-measure.
3. Apply Lever 2 — model routing: send easy questions to a cheap model. Re-measure.
4. Apply Lever 3 — context trimming + streaming: send fewer tokens, and show them sooner. Re-measure.
5. Assemble the before/after dashboard and attribute every dollar and millisecond saved to the lever that saved it.

Each lever you’ve already met in Part 6 as a concept; here you watch it act on a single workload and compound with the others.

Why “memory” still appears in the throughline

You don’t own HBM here — that’s the provider’s cost. Your memory budget is the context window: the finite token allowance you fill on every call. Every token you spend filling it adds prefill latency and dollars, and (as RAG showed) can even hurt quality. Trimming context is this part’s version of “use less memory.”

What you’ll take away

By the end you’ll have a mental dashboard you can put on any LLM feature: the four or five numbers that define its cost and latency, the handful of app-layer levers that move them, and the discipline to measure before and after each change. The levers need no GPUs, no retraining, no infra migration — they are a cache header, a routing if-statement, a reranker, and a streaming call. The cheapest efficiency wins you will ever ship are the ones in this part, and now you’ll watch them land.

The bridge ahead

Next we lay out the plan — the exact app, the exact metrics, and the order we’ll pull the levers — so that every later win is attributable to a single, measured change.

Check your understanding

1. What single question has run through this entire book, and how does the capstone answer it differently from the earlier parts?
2. State the capstone’s goal as a precise, three-part objective.
3. Why does the method insist on building and measuring a baseline before applying any optimization?
4. In an API-based app you don’t own the GPU’s memory — so what plays the role of “memory” in the cost throughline?
5. Why is “make it cheaper by using a smaller model” an incomplete answer to this capstone’s goal?

Show answers

1. What does this cost — in latency, memory, and dollars — and how do we make it cheaper? Earlier parts answered it for systems you own (GPUs, KV cache, batching); the capstone answers it for an app you build on a hosted API, where the levers live outside the model.
2. Cut dollars-per-request and time-to-first-token without sacrificing answer quality. All three clauses are held simultaneously.
3. Because un-measured optimization is superstition — you can’t attribute or even confirm a win you never weighed. A measured baseline lets every later change be credited to a single lever.
4. The context window — the finite per-request token budget you fill, where every token added costs prefill latency and dollars (and can hurt quality).
5. It usually trades cost for quality, violating the “without sacrificing quality” clause. The goal is to cut cost and latency while a real eval keeps quality fixed.