Lever 3: Context Trimming & Streaming

Routing picked a cheaper model; this lever shrinks what we hand it and fixes the blank screen. Two distinct moves bundle here: context trimming attacks the token count (dollars and prefill latency), and streaming attacks the perceived latency (TTFT). They target different numbers, so we measure them separately. The trimming half draws straight on RAG efficiency.

Part A — context trimming

The baseline retrieves top_k=20 chunks “just in case,” stuffing ~8,000 tokens of context that’s mostly irrelevant. Every one of those tokens is paid on every call and dilutes the model’s attention. The fix: retrieve wide for recall, then rerank down to the few that matter.

def build_context(question: str, q_vec) -> str:
    candidates = vector_store.search(q_vec, top_k=20)     # wide net: high recall, cheap
    top = reranker.rerank(question, candidates)[:3]       # narrow: keep the genuine best 3
    return "\n\n".join(h.text for h in top)               # ~1,200 tokens, not ~8,000

def trim_history(history: list) -> list:
    if len(history) > 6:
        return [summarize(history[:-4])] + history[-4:]   # summarize old turns, keep recent
    return history

Three trims, each a token knob:

• top-3 instead of top-20 — context drops from ~8,000 to ~1,200 tokens. A reranker keeps recall high (search wide) while the prompt stays small (send narrow).
• summarize history — replace a long transcript with a short summary plus the recent turns.
• cap max_tokens — every output token decoded is time and dollars, and output costs several× input. A Q&A answer rarely needs 1,024 tokens; cap it.

New token budget: ~600 system + ~1,200 context + ~400 summarized history + ~100 question ≈ 2,300 input tokens (down from 9,600), output capped to ~300.

Trim to the quality floor, not past it

Cutting k too far risks dropping the chunk that actually holds the answer — a wrong answer to save a few tokens. The reranker is the defense (wide recall, narrow send), and the eval set is the floor: tighten k and max_tokens until quality dips, then stop.

Part B — streaming

Trimming made the call cheaper and prefill faster, but the user still waits out the whole decode before seeing anything — because the baseline blocks on the full response. Streaming changes the experience without changing the total work: emit each token as it’s generated, so the user sees content the instant prefill finishes.

def answer_streamed(question: str, context: str):
    with client.messages.stream(
        model=route(question, context),       # routing from Lever 2
        max_tokens=512,                        # capped output — fewer tokens decoded
        system=[
            {"type": "text", "text": SYSTEM_PROMPT},
            {"type": "text", "text": context, "cache_control": {"type": "ephemeral"}},
        ],
        messages=[{"role": "user", "content": f"Question: {question}"}],
    ) as stream:
        for text in stream.text_stream:        # tokens arrive live → user sees output now
            yield text
        final = stream.get_final_message()     # full message + usage when done
    print(final.usage.output_tokens)

Streaming doesn’t reduce end-to-end time much — the same tokens are decoded — but it collapses perceived TTFT from “the whole answer” to “the prefill time.” After caching and trimming, prefill is already tiny (~150–250 ms), so the user now sees the first words almost instantly instead of after several seconds.

Non-streamed:  [ prefill ][ ───── decode whole answer ───── ] → first content
               └──────────── user sees nothing ─────────────┘   (~seconds)

Streamed:      [ prefill ] t1 t2 t3 t4 ... → first content at ~prefill time (~200 ms)
               └ TTFT ┘

Measuring both halves

Token reduction (trimming): input ~9,600 → ~2,300 (a ~4× cut), output ~400 → ~300. But trimming interacts with Lever 1’s caching, and not for free: per-question reranking makes the ~1,200-token context different on every question, so it is no longer a stable, cacheable prefix. The only span that still repeats verbatim across a session is the fixed system + instructions block — and that block must clear the model’s minimum cacheable length to cache at all: 1,024 tokens on Claude Opus 4.8, 4,096 tokens on Claude Haiku 4.5 (a prefix under the floor is silently billed uncached, not cached). So we cache only the stable system+instructions prefix, sized to ~1,100 tokens to clear the Opus minimum, and treat the reranked context, summarized history, and question as uncached. On the 70% Haiku path the whole ~2,300-token prompt sits below Haiku’s 4,096-token floor, so it runs entirely uncached — no real loss, since Haiku’s per-token rate is already ~10× under Opus:

Per Haiku request (uncached — prompt below Haiku's 4,096-token cache floor):
    2,300 × 0.1 + 300 × 0.5 = 230 + 150 = 380 units
Per Opus request (stable ~1,100 prefix cached at 0.1×, ~1,200 volatile uncached):
    1,100 × 0.1 + 1,200 × 1 + 300 × 5 = 110 + 1,200 + 1,500 = 2,810 units
Blended = 0.70 × 380 + 0.30 × 2,810 ≈ 266 + 843 ≈ 1,110 units ≈ 0.10× of baseline

The lesson hiding in that math: trim too far and you fall under the cache floor, so caching and trimming partly cancel instead of stacking. Lever 3 still cuts dollars (fewer uncached tokens on the Opus slice, a tighter output cap), but its headline win is latency, not another big cost drop.

Perceived-latency win (streaming): perceived TTFT drops from the baseline’s ~4,600 ms (full answer) to ~200 ms (prefill only) — a ~20× improvement in felt responsiveness, even though end-to-end only falls to ~2,500 ms p99. Same work, transformed experience.

Scorecard after Lever 3

Metric	Baseline	+ Caching	+ Routing	+ Trim & Stream
Input tokens / request	~9,600	~9,600	~9,600	~2,300
Output tokens / request	~400	~400	~400	~300
Dollars / request	1.00×	~0.33×	~0.12×	~0.10×
Perceived TTFT	~4,600 ms	~3,400 ms	~1,800 ms	~200 ms
Quality (eval pass)	~92%	~92%	~91%	~91%

Trimming kept the dollars falling (now ~0.10× — roughly 10× cheaper than baseline) and streaming did what no cost lever could: made the app feel instant. Quality holds at 91% because the reranker preserves recall. All the levers are pulled — now we assemble the full dashboard and attribute every win.

Under the hood — streaming only exposes your prefill; the cost levers shrink it

Streaming doesn’t make TTFT small — it sets perceived TTFT equal to the time the server takes to emit its first token, which is essentially prefill plus any queue wait. On a raw, uncached, 9,600-token baseline, streaming alone would still leave the user staring at the full ~1,400 ms prefill before the first word. The reason TTFT here collapses to ~200 ms is that the earlier levers already shrank prefill: caching skips re-prefilling the stable prefix, and trimming cut the prompt ~4×. So the latency lever and the cost levers aren’t independent — they compound on TTFT. Streaming converts a small prefill into a near-instant first token; it can’t rescue a fat one.

The architect’s lens

Lever 3 bundles two moves that hit two different numbers — measure them apart:

• Why does it exist? Because the baseline retrieves top_k=20 “just in case” (~8,000 tokens of mostly-irrelevant context) and blocks on the full response, so the user stares at a blank screen.
• What problem does it solve? Trimming attacks the token count — rerank to top-3 (~1,200 tokens), summarize old history, cap max_tokens — cutting input ~4× (9,600→~2,300) to ~0.10× of baseline; streaming attacks perceived TTFT, dropping it to ~200 ms.
• What are the trade-offs? Streaming changes the experience, not the total work (same tokens decoded), and only exposes prefill — so it relies on the earlier levers having shrunk prefill; trimming risks dropping the answer-bearing chunk.
• When should I avoid it? Don’t cut k or max_tokens past the quality floor — the reranker (wide recall, narrow send) and the eval set are the guards; tighten until quality dips, then stop.
• What breaks if I remove it? You pay for ~8,000 tokens of noise per call (dollars and diluted attention), and the user waits out the entire decode before seeing a single word.

Check your understanding

1. Why are context trimming and streaming treated as two separate levers measured against different metrics?
2. How does a reranker let you cut top_k from 20 to 3 without dropping the answer?
3. Streaming barely changes end-to-end time — so what does it change, and why does that matter to the user?
4. Why is capping max_tokens a cost lever, and why does it hit the bill harder per token than trimming input?
5. What’s the failure mode of trimming too aggressively, and what two things guard against it?

Show answers

1. They move different numbers: trimming cuts token count (dollars and prefill latency), while streaming cuts perceived latency (TTFT) without changing token count. Bundling their metrics would hide which one delivered which win.
2. Retrieve a wide candidate set cheaply (high recall) and then re-score it with a sharper reranker, keeping only the genuine best few — so the answer-bearing chunk is still surfaced, but the prompt sent to the model stays small.
3. It changes perceived TTFT — the user sees the first tokens at roughly prefill time instead of after the whole answer is decoded. The total work is the same, but felt responsiveness improves dramatically (here ~4,600 ms → ~200 ms).
4. Every output token is decoded one at a time (time) and billed at the output rate, which is several× the input rate — so capping output saves expensive tokens. Trimming input saves cheaper tokens, though there are more of them.
5. Cutting k or max_tokens too far drops the chunk (or truncates the answer) that actually holds the information — a wrong/incomplete answer to save tokens. Guards: the reranker (wide recall, narrow send) and the eval set (stop tightening when quality dips).