The Baseline: A Naive RAG App

The plan said baseline first, measure before optimizing. So here is the version almost everyone builds on day one: it works, it answers questions correctly, and it is quietly expensive on every single call. We build it, put the meter on it, and read off the numbers that the next three pages will attack.

The naive build

The shortest path from “I have documents” to “it answers questions”: chunk and embed once, then at query time grab a generous pile of chunks, paste them into one big prompt, and hand the whole thing to the strongest model available.

import anthropic

client = anthropic.Anthropic()

SYSTEM_PROMPT = "You answer questions using only the provided context. Cite the source."

# --- index time (once): embed every chunk and store the vectors ---
chunks = split_into_chunks(load_docs())              # ~400 tokens each
vectors = embedder.embed([c.text for c in chunks])   # a separate embedding model
vector_store.upsert(chunks, vectors)

# --- query time (every request) ---
def answer(question: str) -> str:
    q_vec = embedder.embed([question])[0]
    hits = vector_store.search(q_vec, top_k=20)      # naive: grab a lot, just in case
    context = "\n\n".join(h.text for h in hits)      # ~8,000 tokens stuffed in

    resp = client.messages.create(
        model="claude-opus-4-8",                     # one strong model for everything
        max_tokens=1024,
        system=[{"type": "text", "text": SYSTEM_PROMPT}],
        messages=[{
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}",
        }],
    )
    # no streaming — we block until the whole answer is generated
    print(resp.usage.input_tokens, resp.usage.output_tokens)
    return resp.content[0].text

Four naive choices, each a future lever:

• top_k=20 — retrieve a big pile so we “don’t miss anything.” Most of it is irrelevant padding.
• One big stuffed prompt — system prompt + 8,000 tokens of context + the question, rebuilt and re-sent on every call.
• claude-opus-4-8 for everything — the most capable, most expensive model, used even for “what’s the refund window?”
• No caching, no streaming — the stable prefix is re-billed every call, and the user stares at a blank screen until the entire answer is done.

Measuring the baseline

The usage object on the response is the meter — no estimation needed. A typical request:

input  tokens:  ~9,600   (600 system + 8,000 context + ~900 history + ~100 question)
output tokens:    ~400

Turn that into cost with the two facts from Part 6: every input token is paid on every call, and output tokens cost several× input. Using illustrative relative units (Opus input = 1 unit/token, output ≈ 5 units/token; always check current pricing):

input  cost ≈ 9,600 × 1 = 9,600 units
output cost ≈   400 × 5 = 2,000 units
            ────────────────────────
per request ≈ 11,600 units      ← we'll call this 1.00× (the baseline index)

Input tokens are 83% of the bill here, and ~8,000 of those 9,600 are retrieved context — most of it never needed to answer the question. That is the waste, in one number.

The latency picture

Two costs, both bad:

prefill 9,600 prompt tokens   ≈ 1,400 ms   (TTFT if we streamed — but we don't)
decode  400 output tokens     ≈ 3,200 ms
                              ─────────────
end-to-end                    ≈ 4,600 ms

Because there is no streaming, the user sees nothing for the full ~4,600 ms — the perceived time-to-first-content is the end-to-end time. A big prompt makes prefill slow, and a non-streamed call makes the user wait out the entire decode on top of it. Both halves of the latency throughline — and the dollars — are inflated at once.

The context window is the hidden memory cost

Those 9,600 input tokens are this app’s “memory” footprint: the slice of the context window we fill every call. It’s not HBM we own, but we pay for it in prefill latency and dollars on every request — and stuffing 20 chunks risks the lost-in-the-middle effect, so the padding can lower quality too.

The baseline scorecard

Metric	Baseline value
Input tokens / request	~9,600
Output tokens / request	~400
Dollars / request	1.00× (index)
Perceived TTFT	~4,600 ms
End-to-end p99	~7,000 ms
Quality (eval pass)	~92%

It works — 92% on the eval set — and that’s the trap: a working app hides a recurring overpayment. We hold that 92% fixed and go after everything else.

Where the waste is

Three leaks, each matched to a lever:

1. Re-billed prefix. The system prompt and the session’s document context repeat call after call, re-prefilled and re-billed every time. → Prompt caching.
2. Over-powered model. Opus answers the easy 70% of questions that a far cheaper model would nail. → Model routing.
3. Too much context, no streaming. 20 chunks where 3 would do, and a blank screen for 4.6 s. → Context trimming + streaming.

We start with the safest, most mechanical win: caching the prefix.

Under the hood — why output tokens cost several× input

The price gap mirrors a real compute asymmetry from prefill vs decode. The ~9,600 prompt tokens are prefilled in one parallel, compute-bound pass — the GPU crunches the whole prompt at once, which is why 9,600 input tokens take only ~1,400 ms. The ~400 output tokens are decoded one at a time: each step reloads the model’s weights and the growing KV cache from memory to emit a single token, so decode is memory-bandwidth-bound and strictly sequential — which is why far fewer tokens (~400) still take more than twice as long (~3,200 ms) as prefilling 24× as many. The provider prices output higher because each output token is genuinely more expensive to produce. That’s why every later lever that touches output — capping max_tokens, routing to a faster model — moves the bill and the clock harder per token than trimming input does.

Check your understanding

1. List the four naive choices in the baseline build, and which lever each one sets up.
2. From the worked numbers, what fraction of the per-request bill is input tokens, and what is most of that input?
3. With no streaming, what does the user actually perceive as the time-to-first-content, and why?
4. Why is reporting a baseline quality score (92%) essential before optimizing anything?
5. Identify the three sources of waste in the baseline and the lever that targets each.

Show answers

1. top_k=20 (→ context trimming), one big stuffed prompt re-sent every call (→ prompt caching of the stable prefix), claude-opus-4-8 for everything (→ model routing), and no caching/no streaming (→ caching + streaming).
2. Input tokens are ~9,600 × 1 = 9,600 of ~11,600 units ≈ 83% of the bill, and ~8,000 of those input tokens are retrieved context — mostly irrelevant padding.
3. The full end-to-end time (~4,600 ms), because a non-streamed call returns nothing until the entire answer is generated — the user waits out prefill and the whole decode before seeing a single token.
4. So that later “cheaper” changes can be checked against it — without a fixed quality baseline, a cost win and a quality regression look the same in the cost numbers. It’s the guardrail that defines “without sacrificing quality.”
5. Re-billed stable prefix (→ prompt caching), an over-powered model on easy questions (→ model routing), and too much context with no streaming (→ context trimming + streaming).