Reasoning Models & Test-Time Compute

The book opened by splitting cost into training vs inference: training is a big one-time bill, inference is the per-call meter you pay forever. For years the assumption was that, once trained, a model spends a roughly fixed amount of compute per answer. The 2024–2025 reasoning-model wave broke that assumption. It made inference compute a dial: let the model think longer — generate a long internal chain-of-thought before answering — and accuracy on hard problems goes up. This is test-time compute, and it turns “how smart is the model?” partly into “how much are you willing to spend per answer?”

The core idea: spend compute now to buy accuracy

A standard model reads the prompt and starts answering immediately. A reasoning model first generates a long stretch of intermediate reasoning — working through the problem step by step, considering approaches, checking itself — and only then emits the final answer. That intermediate reasoning is more tokens, and tokens are compute and dollars. The bet is that on genuinely hard tasks (math, code, multi-step logic), the extra thinking pays for itself in correctness.

   standard model:   prompt ─▶ answer                       (cheap, fast)

   reasoning model:  prompt ─▶ [ long internal reasoning ] ─▶ answer
                                ↑ you pay for these tokens,
                                  often without seeing them

Chain-of-thought prompting (Wei et al., 2022) showed that asking a model to “think step by step” improved reasoning. Reasoning models internalize that: they are trained to produce long, structured chains of thought on their own, and to spend more of them on harder problems.

The 2024–2025 reasoning-model wave

The visible turning point was OpenAI’s o1, released as a preview in September 2024 — a model explicitly built to “think before it answers” and to scale accuracy with how long it thinks. Through late 2024 and into 2025, the rest of the field followed with their own reasoning models and “extended thinking” modes. Anthropic, Google, DeepSeek, and others shipped models or modes that spend extra inference compute on a long chain-of-thought before responding.

Dates and drift

o1’s preview is September 2024 — that anchor is solid. Everything about which model is most accurate or cheapest at reasoning drifts month to month; treat any such ranking as a snapshot. The durable fact is the mechanism: these models trade more inference tokens for more accuracy, and most expose some way to control how much thinking they do.

The thinking tokens you pay for

Here is the part that matters for a cost-conscious engineer: you are billed for the reasoning tokens even when you never see them. Many reasoning models hide or summarize the raw chain-of-thought, but those hidden tokens were still generated, still consumed GPU time, and still appear on your bill — typically priced as output tokens, which carry the ~5× premium over input.

By the numbers — what thinking costs

Suppose a hard question would take a standard model 200 output tokens to answer. A reasoning model produces, say, 2,000 tokens of internal chain-of-thought plus the same 200-token answer. Pricing in the relative units the book has used (input = 1, output ≈ 5):

standard:   200 output × 5  =  1,000 units
reasoning: 2,200 output × 5  = 11,000 units

The reasoning answer costs about 11× more for this question — entirely in output-priced thinking tokens. If that extra spend turns a wrong answer into a right one on a task where correctness is worth it (a contract clause, a migration script, a medical triage), it can be a bargain. If the question was “what’s the capital of France,” you just paid 11× for nothing. The whole skill is knowing which is which.

It also costs latency: those 2,000 thinking tokens are generated one at a time, decode-bound, so the user waits through all of them before the first word of the answer appears. Reasoning models trade time-to-answer for quality-of-answer.

When test-time scaling is worth it

Spending more inference compute is not free accuracy — it is accuracy you rent, per call, forever. A useful way to decide:

Spend more thinking when…	Don’t, when…
the task is genuinely hard (multi-step math, code, proofs)	the task is simple lookup or formatting
a wrong answer is expensive (correctness ≫ cost)	high volume, low stakes, latency-sensitive
you can’t easily verify the answer cheaply yourself	a cheap verifier or schema check suffices
the workload is low-volume	the workload runs millions of times a day

This is the same logic as model routing, one level up: there, you chose which model; here, you also choose how hard it thinks. Many systems combine both — route easy traffic to a cheap non-reasoning model, and reserve expensive extended thinking for the requests a classifier flags as hard. Research on scaling test-time compute (e.g. Snell et al., 2024) made the trade explicit: for a fixed accuracy target there is an optimal amount of inference compute to spend, and overspending buys little.

Test-time compute is a knob, not a setting

The mistake is treating “reasoning mode” as always-on or always-off. It is a dial you set per request. The throughline question — what does this cost in latency, memory, and dollars, and how do we make it cheaper? — gets a new answer here: the cheapest reasoning is the reasoning you only pay for on the requests that actually need it.

Distilling it back into a cheap model

There is a second way to make reasoning cheaper: don’t pay for it at inference at all — bake it in ahead of time. Use a large, slow reasoning model to generate high-quality chains of thought, then distill that behavior into a smaller, faster model that imitates the reasoning without re-deriving it from scratch each time. Several 2024–2025 releases shipped distilled reasoning models for exactly this reason: a smaller model that has absorbed much of the big model’s reasoning skill, so you pay the test-time-compute bill once during training instead of on every call.

The trade-off is the familiar one from the compression part: distillation moves cost from inference (recurring) to training (one-time), and a distilled student rarely matches the full reasoning model on the very hardest problems. So the two strategies coexist: distill the common reasoning into a cheap model for everyday volume, and keep a full test-time-scaling model in reserve for the genuinely hard tail.

It actually happened — o1, September 2024

OpenAI released o1 as a preview in September 2024, described as a model that produces a long internal chain-of-thought before answering and improves on hard reasoning benchmarks as it is allowed to think longer. It was the moment “test-time compute” went from a research idea to a product category, and it kicked off the reasoning-model wave that defined the following year. It also made the cost story concrete and a little uncomfortable: the chain-of-thought was largely hidden from the user, yet billed — the clearest example yet that with reasoning models, you pay for tokens you cannot read. The lesson for an efficiency-minded engineer: when thinking is metered, decide deliberately how much thinking each request is worth.

The architect’s lens

Reasoning turns inference compute into a dial — and the dial is metered:

• Why does it exist? Because letting a model generate a long internal chain-of-thought before answering raises accuracy on genuinely hard problems — test-time compute breaks the old assumption that a trained model spends fixed compute per answer.
• What problem does it solve? Hard multi-step tasks (math, code, proofs) where extra thinking pays for itself in correctness — accuracy you can rent per call by spending more.
• What are the trade-offs? You’re billed for the hidden thinking tokens at the ~5× output rate (2,000 thinking + 200 answer ≈ 11× the cost of a 200-token reply) and the user waits through all of them — time and dollars traded for quality.
• When should I avoid it? On simple lookups/formatting, and on high-volume latency-sensitive traffic where the 11× cost and delay multiply across millions of calls — treat reasoning as a per-request knob, not always-on.
• What breaks if I remove it? You lose the accuracy gains on the hard tail; alternatively, distill the common reasoning into a cheap model to pay the bill once in training and keep a full reasoning model only for the genuinely hard minority.

Check your understanding

1. What does “test-time compute” mean, and how does it change the old assumption that a trained model spends a fixed amount of compute per answer?
2. Why can a reasoning model’s bill be many times larger than a standard model’s for the same final answer — even when you never see the extra tokens?
3. Give two situations where spending more on thinking tokens is worth it, and two where it isn’t.
4. How is choosing a reasoning model’s “thinking budget” related to model routing?
5. What does distilling reasoning into a smaller model trade away, and what does it buy?

Show answers

1. It means accuracy can be increased by letting the model generate more inference-time reasoning tokens before answering; compute per answer becomes a dial you set, not a fixed amount, so “how good is the answer” depends partly on how much you spend per call.
2. The model generates a long internal chain-of-thought (often hidden or summarized) before the final answer; those thinking tokens are still generated and billed — typically at the higher output-token rate — so the per-question cost can be roughly an order of magnitude higher.
3. Worth it: genuinely hard multi-step tasks, and cases where a wrong answer is expensive or hard to verify cheaply. Not worth it: simple lookups/formatting, and high-volume latency-sensitive workloads where the extra cost and delay multiply across millions of calls.
4. Both match effort to difficulty: routing chooses which model handles a request, while the thinking budget chooses how hard the model thinks; many systems combine them, reserving expensive extended thinking for requests a classifier flags as hard.
5. It trades away some peak accuracy on the very hardest problems (the small student rarely matches the full reasoning model) and moves cost from recurring inference to one-time training; it buys much cheaper, faster reasoning on everyday volume because you paid the thinking cost once during distillation.