Model Routing & Cascades

RAG shrank what you send. Routing changes who you send it to. The premise is blunt: most requests in a real workload are easy, and you are almost certainly paying frontier-model prices to answer “what’s 2+2?” with the same model you use for a hard multi-step proof. The throughline question — fewer dollars per call — has no bigger single lever than this, because model prices span the better part of an order of magnitude. Match the model to the difficulty and the blended cost collapses.

The Claude price ladder

cheap, fast  ◀──────────────────────────────▶  capable, expensive
  claude-haiku-4-5   claude-sonnet-4-6   claude-opus-4-8   (claude-fable-5)

Haiku is the cheapest and fastest; Sonnet is the balanced middle; Opus is the most capable and most expensive (with Fable above it for the hardest reasoning). Haiku is roughly 5× cheaper per token than Opus — and about an order of magnitude cheaper than Fable at the top of the ladder — so every request you don’t send to the expensive tier is a large, recurring saving. (Always check current pricing.)

Two patterns: upfront router and cascade

Upfront router. A cheap classifier (a small model, or even keyword/heuristic rules) looks at the request and decides which model to send it to — before any expensive call happens.

request ─▶ router ─┬─ "easy"   ─▶ Haiku
                   ├─ "medium" ─▶ Sonnet
                   └─ "hard"   ─▶ Opus

Cascade. Try the cheap model first; check whether its answer is good enough; escalate only if not.

request ─▶ Haiku ─▶ confident & valid? ─yes─▶ return
                          │
                          no ─▶ Sonnet ─▶ good enough? ─no─▶ Opus

The cascade needs a verification step — a confidence score, a schema/validity check, a self-grade, or a cheap verifier model — to decide whether to accept or escalate. The router needs a good upfront difficulty classifier. Both aim at the same thing: keep the expensive model for the requests that genuinely need it.

Note

A cascade pays the cheap call and then the expensive one on escalated requests, so it’s a bet that escalations are rare. If most requests escalate, you’ve added cost, not saved it. Routing pays off only when the cheap tier actually handles the bulk of traffic.

Worked example: the blended cost

Say a workload is 80% easy, 15% medium, 5% hard. Using illustrative relative per-call costs — Haiku = 1, Sonnet ≈ 3, Opus ≈ 10 (order-of-magnitude spread; check real pricing):

Everything to Opus:

cost = 1.00 × 10 = 10.0 units/call

Routed by difficulty:

cost = 0.80 × 1   (Haiku)
     + 0.15 × 3   (Sonnet)
     + 0.05 × 10  (Opus)
     = 0.80 + 0.45 + 0.50 = 1.75 units/call

Routing cuts the blended cost from 10.0 to 1.75 — about 83% cheaper — while the hard 5% still gets Opus-grade answers. The cheap tier carries the volume; the expensive tier carries the difficulty. (A cascade’s math is similar but adds the cheap probe cost on escalated calls — still a large net win when escalations are the minority.)

The risk: mis-routing

Routing’s whole danger is sending a hard request to a model that botches it. A request the router mislabels as easy gets a wrong answer at a discount — false economy. So routing is only as safe as its routing decision.

Guard it:

• Evals. Measure quality per tier on real traffic before trusting the split. Know your cheap tier’s actual accuracy on the work you’ll route to it.
• Verification in cascades. A solid “is this good enough?” check turns mis-routing into an escalation (extra cost) rather than a wrong answer (worse).
• Tune the threshold. Escalate more readily for high-stakes requests; accept the cheap answer more readily for low-stakes ones.

Route on difficulty, not on hope

“This is probably easy” is not a routing strategy. Either classify difficulty with something you’ve measured, or run a cascade with a real verification gate. An unverified cheap-first cascade silently ships the cheap model’s mistakes.

Where this fits

Routing composes with everything before it: a routed request can still be lean, cached, and RAG-trimmed. It’s Lever 2 in the capstone, where it’s applied to a real workload and measured against the others. The instinct to carry out of this page: the most expensive model is a resource, not a default. Spend it only where the difficulty earns it.

It actually happened — FrugalGPT and learned routers (2023–2024)

Routing isn’t folklore; it’s a documented research result. In 2023, the FrugalGPT paper (Chen, Zaharia, and Zou at Stanford) formalized the LLM cascade — query a cheap model first, score its answer with a lightweight verifier, and escalate to a stronger model only on low confidence — and reported large cost reductions while matching a frontier model’s quality on many queries. In 2024, RouteLLM (from the LMSYS/Berkeley group) pushed the upfront-router idea: instead of hand-written rules, train a small classifier on preference data to predict whether the cheap or the strong model will satisfy a given query, and route accordingly. Both lines of work land on the throughline this page argues — the expensive model is a resource you spend only when the difficulty earns it — and both make the same warning concrete: the saving is real only if your verifier or router is actually accurate on your traffic, which is why they ship with evals, not vibes.

The architect’s lens

Routing is the biggest single lever on dollars-per-call — and the easiest to get quietly wrong:

• Why does it exist? Because model prices span ~an order of magnitude (Haiku ≈ 5× cheaper than Opus, ≈ 10× cheaper than Fable) yet most real traffic is easy — paying frontier prices to answer “what’s 2+2?” is pure waste.
• What problem does it solve? Blended cost: on an 80/15/5 easy/medium/hard split, routing cuts the per-call cost from 10.0 (all-Opus) to ~1.75 units — about 83% cheaper — while the hard 5% still gets Opus.
• What are the trade-offs? A router needs an accurate upfront difficulty classifier; a cascade needs a real verification gate and pays the cheap probe before escalating — so if most requests escalate, you’ve added cost, not saved it.
• When should I avoid it? When the hard fraction is large (or the cheap tier can’t really handle the “easy” work) so most traffic lands on the expensive model anyway, and when you can’t measure tier quality on your own traffic.
• What breaks if I remove it? You default every request to the most capable model — correct but needlessly expensive; route on hope instead of measured difficulty and you ship the cheap model’s mistakes at a discount (mis-routing).

Check your understanding

1. What’s the core assumption that makes model routing save money, and when does it fail?
2. Explain the difference between an upfront router and a cascade, and what extra component each needs.
3. With an 80/15/5 easy/medium/hard split and relative costs Haiku=1, Sonnet=3, Opus=10, compute the routed blended cost and compare to all-Opus.
4. Why can a cascade sometimes cost more than just using one model?
5. What is mis-routing, why is it dangerous, and what two guards reduce it?

Show answers

1. That most requests in the workload are easy and can be handled well by a cheap model, so only a minority need the expensive one. It fails when the hard fraction is large (or the cheap tier can’t actually handle the “easy” work), so most traffic ends up on the expensive model anyway.
2. An upfront router classifies difficulty before any expensive call and sends each request directly to the right tier; it needs a good difficulty classifier. A cascade tries the cheap model first and escalates only on failure; it needs a verification/confidence step to decide whether to accept or escalate.
3. Routed = 0.80×1 + 0.15×3 + 0.05×10 = 0.80 + 0.45 + 0.50 = 1.75 units/call, vs 10.0 for all-Opus — about 83% cheaper.
4. A cascade pays the cheap call and then the expensive call on every escalated request; if escalations are frequent, the added probe cost outweighs the savings.
5. Mis-routing is sending a request to a model too weak for it (e.g., labeling a hard request “easy”), yielding a wrong answer at a discount — a false economy. Guards: evals that measure each tier’s real quality on the routed traffic, and a verification gate in cascades that converts a bad cheap answer into an escalation rather than a shipped error.