Lever 1: Prompt Caching

The baseline re-sends the same system prompt and the same document context on every call, paying full price to re-prefill bytes that never changed. That is the cleanest waste to kill, and the safest: caching the stable prefix changes the bill, not the answer. This is the application-layer face of the prefix caching you met in Part 3 — the same KV-reuse mechanism, now exposed as an API knob you control without owning a GPU.

Why a prefix is cacheable

A transformer is causal: a token’s KV depends only on the tokens before it. So if many requests begin with the exact same span — the same system prompt, the same instructions, the same document context — that span’s prefill work is bit-for-bit identical every time. Recomputing it is pure waste. In a doc-Q&A session the user typically asks several questions about the same documents, so the big context block is the same across turns. That’s a fat, hot, repeated prefix begging to be cached.

turn 1:  [ system + document context ] + "What's the refund window?"
turn 2:  [ system + document context ] + "Does it cover digital goods?"
turn 3:  [ system + document context ] + "Who do I email?"
         └────── identical prefix ──────┘   └──── unique tail ────┘

Enabling it

On the Messages API you mark the end of the stable prefix with cache_control. Render order is tools → system → messages, so stable content goes first (in system) and the volatile question goes last (in messages), after the breakpoint:

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {"type": "text", "text": SYSTEM_PROMPT},
        {
            "type": "text",
            "text": session_document_context,         # stable across the session's turns
            "cache_control": {"type": "ephemeral"},   # ← cache everything up to here
        },
    ],
    messages=[{"role": "user", "content": f"Question: {question}"}],  # volatile: after the breakpoint
)

print(resp.usage.cache_creation_input_tokens)  # first call: wrote the cache (~1.25× write)
print(resp.usage.cache_read_input_tokens)      # later calls: read it (~0.1× a normal token)
print(resp.usage.input_tokens)                 # only the uncached tail (the question)

Verify it works. If cache_read_input_tokens stays zero across repeated calls, a silent invalidator is at work — a datetime.now() in the system prompt, a per-request ID before the breakpoint, non-deterministic JSON ordering. Any byte change before the breakpoint invalidates the whole cached prefix.

What’s cacheable, and the conditions

• Cacheable: the frozen system prompt, tool definitions, few-shot blocks, and the document context that persists across a session’s turns — anything stable and repeated.
• Not cacheable: the user’s question, per-request timestamps, anything that changes every call. Keep it after the breakpoint.
• Hit conditions: an exact, token-level match of the prefix, on the same model (caches are model-scoped), within the TTL.
• TTL: the cache entry expires after a short window (commonly ~5 minutes; a longer 1-hour option exists at a higher write premium). The win is real only when calls reuse the prefix inside that window — which a live session does.

Caches are per-model

A cached Opus prefix does nothing for a request routed to Haiku — switching models means a cold cache. This matters in the next lever, where routing and caching interact: each model warms its own prefix within its share of traffic.

Before / after on cost

Of the baseline’s ~9,600 input tokens, roughly 8,600 are stable prefix (system + the session’s document context) and ~1,000 vary (recent history tail + question). The economics, in the same relative units as the baseline (write ≈ 1.25×, read ≈ 0.1×; output unchanged):

Baseline (no cache), per call:
  input  9,600 × 1          = 9,600
  output   400 × 5          = 2,000        total ≈ 11,600 units   (1.00×)

With caching, on a cache hit:
  cached prefix  8,600 × 0.1 =   860
  uncached tail  1,000 × 1   = 1,000
  output           400 × 5   = 2,000
                             ───────────
                         total ≈ 3,860 units  ≈ 0.33× of baseline

The first call pays a one-time write premium (8,600 × 1.25 on top of the tail); every reused call after that pays the 0.1× read rate. Across a multi-turn session the write amortizes to nothing and the prefix bills at roughly a tenth — a ~3× cut to the whole per-request bill, driven entirely by input tokens.

And the latency

Skipping the prefix prefill is a TTFT win too: the request now only prefills the ~1,000 varying tokens instead of 9,600.

prefill:  ~1,400 ms  →  ~200 ms   (prefix prefill skipped on a hit)
perceived TTFT (still no streaming): ~4,600 ms → ~3,400 ms

We’re still not streaming, so the user still waits out the decode — but the cheapest token is the one you never compute, and we just stopped computing ~8,600 of them per call.

Scorecard after Lever 1

Metric	Baseline	+ Caching
Input tokens billed / request	~9,600	~1,860 (on a hit)
Output tokens / request	~400	~400
Dollars / request	1.00×	~0.33×
Perceived TTFT	~4,600 ms	~3,400 ms
Quality (eval pass)	~92%	~92%

Quality is untouched — caching reuses identical KV, so the answer is exactly what it would have been. A 3× cost cut and a faster first token, at zero quality risk. For the deeper mechanism see prefix & prompt caching and the layered view in caching strategies. Next we attack the model itself: most of these questions never needed Opus. → Model routing.

The architect’s lens

Lever 1 is the safest cut in the capstone — it changes the bill, not the answer:

• Why does it exist? Because the baseline re-sends the same system prompt and document context every turn, paying full price to re-prefill bytes that never change — and a transformer’s causal KV makes that fixed prefix bit-for-bit identical each call.
• What problem does it solve? Repeated input tokens: marking the stable prefix with cache_control bills the ~8,600-token prefix at ~0.1× on a read, cutting the per-request bill ~3× (to ~0.33× of baseline) and skipping its prefill (~1,400 ms → ~200 ms).
• What are the trade-offs? A one-time ~1.25× write premium, a short ~5-min TTL, and caches are per-model — a single changed byte before the breakpoint (a datetime.now(), a per-request ID) silently invalidates everything.
• When should I avoid it? When calls don’t reuse the prefix inside the TTL, or when the “stable” content actually varies per request — there’s nothing to cache.
• What breaks if I remove it? Every request re-prefills the full ~9,600 tokens at the 1× rate — and because caching reuses identical KV, removing it changes cost and latency but never quality.

Check your understanding

1. What property of transformers makes a shared prefix’s prefill identical across calls, and why does that make it cacheable?
2. Why must the document context go in system (before the breakpoint) and the question in messages (after it)?
3. How do you confirm from the usage object that caching is actually hitting, and what does a persistent zero there indicate?
4. Using the worked units, why does caching cut the per-request bill roughly 3× even though output tokens didn’t change?
5. Why does prompt caching carry essentially zero quality risk, unlike the later levers?

Show answers

1. Causality — a token’s key/value depend only on preceding tokens — so an identical leading span produces bit-for-bit identical KV every time, regardless of how the requests later differ. That identical prefill can be computed once and reused.
2. Caching is a prefix match keyed on exact bytes from the start; the stable content must come first so it stays byte-identical and cacheable, and the volatile question must come after the breakpoint so it doesn’t change (and thus invalidate) the cached prefix.
3. Check usage.cache_read_input_tokens — a positive value on repeat calls means hits. A persistent zero means a silent invalidator is changing the prefix bytes (a timestamp, a per-request ID, non-deterministic serialization) before the breakpoint.
4. Because input tokens are ~83% of the baseline bill, and caching drops the ~8,600-token stable prefix from a 1× rate to ~0.1×. Even with output unchanged, shrinking the dominant input cost takes the total from ~11,600 to ~3,860 units (~0.33×).
5. Because it reuses identical KV for an identical prefix — the model produces exactly the answer it would have without the cache. Routing (mis-routing) and trimming (missing a chunk) can change the answer; caching cannot.