Lever 2: Model Routing & Cascades

Caching cut what we re-send; routing changes who we send it to. The baseline answers “what’s the refund window?” with claude-opus-4-8 — the most capable, most expensive model — the same one it uses for a hard multi-step inference. Model prices span the better part of an order of magnitude, so matching the model to the difficulty is the single biggest lever on dollars-per-request in this whole capstone. This is Lever 2 from Part 6, now applied to our workload.

The price ladder

cheap, fast  ◀──────────────────────────────────▶  capable, expensive
  claude-haiku-4-5    claude-sonnet-4-6    claude-opus-4-8

Haiku is the cheapest and fastest; Sonnet is the balanced middle; Opus is the most capable and most expensive. Haiku is severalfold cheaper per token than Opus — the exact multiple moves with every pricing revision, so always check current pricing; the worked math below rounds it to ~10× for clean numbers. So every request you don’t send to Opus is a large, recurring saving — if a cheaper model can actually answer it.

Two patterns

Upfront router — a cheap classifier decides the tier before any expensive call:

def route(question: str, context: str) -> str:
    difficulty = classify(question)            # keyword rules, or one cheap Haiku call
    if difficulty == "easy":
        return "claude-haiku-4-5"
    if difficulty == "hard":
        return "claude-opus-4-8"
    return "claude-sonnet-4-6"

def answer(question: str, context: str) -> str:
    model = route(question, context)
    resp = client.messages.create(
        model=model,
        max_tokens=1024,
        system=[
            {"type": "text", "text": SYSTEM_PROMPT},
            {"type": "text", "text": context, "cache_control": {"type": "ephemeral"}},
        ],
        messages=[{"role": "user", "content": f"Question: {question}"}],
    )
    return resp.content[0].text

Cascade — try cheap first, escalate only if a check fails:

def answer_cascade(question: str, context: str) -> str:
    resp = client.messages.create(model="claude-haiku-4-5", max_tokens=1024,
                                  system=SYSTEM, messages=msg(question, context))
    if passes_check(resp):                     # confidence, schema, or a self-grade
        return resp.content[0].text
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024,  # escalate
                                  system=SYSTEM, messages=msg(question, context))
    return resp.content[0].text

The router needs a good difficulty classifier; the cascade needs a real verification step (a confidence score, a validity check, a cheap self-grade). Both aim at the same thing: keep Opus for the questions that genuinely need it.

A cascade is a bet on rare escalations

A cascade pays the cheap call and then the expensive one whenever it escalates. It wins only if escalations are the minority — if most questions escalate, you’ve added the probe cost on top of Opus and made things worse. Measure your escalation rate before trusting it.

The blended-cost math

Say our eval traffic is 70% easy, 30% hard — easy routes to Haiku, hard to Opus. Using the same relative units (Opus input = 1, output = 5; Haiku ≈ 10× cheaper, so input = 0.1, output = 0.5) and keeping caching on from Lever 1 (cached prefix at the model’s own 0.1× read rate):

Per Haiku request (cached prefix 8,600, tail 1,000, output 400):
  (8,600 × 0.1 + 1,000) × 0.1   +   400 × 0.5
   ────────── input ──────────       ─ output ─
        = 186                    +     200      = 386 units

Per Opus request (the Lever-1 cached number):  ≈ 3,860 units

Blended = 0.70 × 386 + 0.30 × 3,860
        = 270 + 1,158
        ≈ 1,428 units  ≈ 0.12× of the 11,600-unit baseline

Routing on top of caching takes us from ~0.33× to ~0.12× — the cheap tier carries the volume, the expensive tier carries the difficulty. Haiku is also faster, so the 70% of traffic on it has markedly lower latency, pulling the blended perceived TTFT down further (to ~1,800 ms, still pre-streaming).

Guard it with the eval set

Routing’s whole danger is mis-routing: a hard question mislabeled “easy” gets a wrong answer at a discount — false economy. The defense is the quality column from the plan:

• Measure quality per tier on real traffic before trusting the split — know Haiku’s actual accuracy on the work you’ll route to it.
• Use a verification gate in cascades so a weak cheap answer becomes an escalation (extra cost) rather than a shipped error (worse).
• Tune the threshold by stakes — escalate readily for high-stakes questions, accept the cheap answer for low-stakes ones.

On our eval set the routed app scores ~91% — one point under the 92% baseline, because a few borderline questions slip to Haiku. That’s inside our tolerance; if it dipped further, we’d tighten the classifier or lower the cascade’s accept threshold.

Route on measured difficulty, not on hope

“This is probably easy” is not a routing strategy. Either classify difficulty with something you’ve measured, or run a cascade with a real verification gate. An unverified cheap-first cascade silently ships the cheap model’s mistakes — a cost win the eval set would have caught.

Scorecard after Lever 2

Metric	Baseline	+ Caching	+ Routing
Dollars / request	1.00×	~0.33×	~0.12×
Perceived TTFT	~4,600 ms	~3,400 ms	~1,800 ms
End-to-end p99	~7,000 ms	~5,000 ms	~4,000 ms
Quality (eval pass)	~92%	~92%	~91%

We’ve gone from 1.00× to ~0.12× — roughly 8× cheaper — while the hard 30% still gets Opus-grade answers and quality holds within a point. The most expensive model is now a resource, not a default. Next we shrink the prompt itself and fix the blank-screen latency: context trimming + streaming.

The architect’s lens

Lever 2 attacks the model itself — the biggest dollars-per-request lever in the build:

• Why does it exist? Because the baseline answers “what’s the refund window?” with Opus, the most expensive model, and model prices span the better part of an order of magnitude — every request you don’t send to Opus is a recurring saving.
• What problem does it solve? Blended cost: on a 70/30 easy/hard split, routing easy to Haiku and hard to Opus (with Lever-1 caching kept on) drops the per-request cost from ~0.33× to ~0.12× of baseline — roughly 8× cheaper than the start — and pulls perceived TTFT to ~1,800 ms.
• What are the trade-offs? A router needs a measured difficulty classifier; a cascade pays the cheap probe then Opus on escalation, so it only wins if escalations are the minority — and caches are per-model, so each model warms its own prefix.
• When should I avoid it? When you can’t measure tier quality on real traffic — routing on hope silently ships the cheap model’s mistakes (mis-routing); the eval set caught the 1-point dip to ~91% here.
• What breaks if I remove it? You send every request to Opus — correct but ~8× more expensive than the routed app, paying frontier prices for the 70% of questions Haiku handles fine.

Check your understanding

1. What core assumption makes routing save money, and when does it stop working?
2. Compute the blended cost for a 70% Haiku / 30% Opus split given per-request costs of 386 (Haiku) and 3,860 (Opus) units. Compare to all-Opus-with-caching (~3,860).
3. Why can a cascade end up more expensive than just calling one model?
4. What is mis-routing, and what two guards keep it from shipping wrong answers?
5. Caches are model-scoped — what does that imply for how caching and routing interact?

Show answers

1. That most requests are easy and a cheap model handles them well, so only a minority need the expensive one. It fails when the hard fraction is large (or the cheap tier can’t actually handle the “easy” work), so traffic ends up on the expensive model anyway.
2. Blended = 0.70 × 386 + 0.30 × 3,860 = 270 + 1,158 ≈ 1,428 units, versus ~3,860 for all-Opus-with-caching — roughly 2.7× cheaper again, and ~0.12× of the original baseline.
3. A cascade pays the cheap probe and then the expensive call on every escalation; if escalations are frequent, the added probe cost outweighs the savings and the total exceeds just using the strong model directly.
4. Mis-routing is sending a request to a model too weak for it (labeling a hard question “easy”), producing a wrong answer at a discount. Guards: per-tier evals measuring real quality on routed traffic, and a verification gate in cascades that turns a bad cheap answer into an escalation rather than a shipped error.
5. A cached prefix only helps requests sent to the same model — routing to a different model means a cold cache there. So each model warms its own copy of the prefix within its share of traffic; the two levers compose but don’t share cache entries.