Speculative Decoding

Everything so far raised throughput — more requests through the same GPU. Speculative decoding attacks latency — how fast one request finishes — and it does so by exploiting the very fact we have leaned on throughout this part: decode is memory-bound. Each big-model decode step reads gigabytes of weights from HBM to produce a single token, leaving the compute units mostly idle. That idle compute is the opportunity. The throughline question — how do we make a token cheaper in latency? — gets an unusually elegant answer here: produce several tokens per expensive weight-read, and prove they are correct for free.

The asymmetry that makes it work

Recall two facts from Part 2:

• Decode processes one token at a time and is bottlenecked by reading weights, so a step that produces 1 token and a step that checks 4 tokens cost almost the same in HBM traffic. The marginal tokens ride along on compute that was idle anyway.
• Prefill processes many tokens in parallel in a single forward pass, and is comparatively cheap per token because it is compute-bound, not starved on bandwidth.

So a big model can verify a batch of k candidate tokens in essentially one prefill-like forward pass — nearly the cost of generating a single token. If only we had candidates to check. That is what the draft model provides.

Draft, then verify

Pair the large target model with a small, cheap draft model (often 10–20× fewer parameters, or even a few-layer head). The loop:

1. The draft model autoregressively generates k proposed tokens. Cheap, because it is small — its weights are a fraction of the target’s, so its memory-bound steps are fast.
2. The target model runs one forward pass over all k proposals in parallel, producing its own probability for each position.
3. Verify left to right: accept each proposed token with a probability derived from comparing the draft’s and target’s distributions. Accept a run of tokens until the first rejection.
4. At the first rejection, resample that one token from a corrected distribution (and discard the rest), then loop. If all k are accepted, a bonus token comes from the target’s final position.

Draft (small, sequential):     d1  d2  d3  d4        ← proposes k = 4
Target (big, ONE parallel pass): [ verify d1 d2 d3 d4 in a single forward ]
                                    ✓   ✓   ✓   ✗
Accept d1, d2, d3 ; resample token 4 from target ; resume drafting
Result: one big-model pass yielded ~3–4 tokens instead of 1

It is lossless — this is the crucial part

The accept/resample rule (rejection sampling against the target’s distribution) is constructed so the emitted tokens have exactly the same distribution as if the target model had generated them alone. Speculative decoding is not an approximation and does not degrade quality — it produces the same output the big model would have, just in fewer sequential steps. The draft model’s only job is to guess well; when it guesses wrong, the math throws the guess away.

The win, quantified

Let α be the acceptance rate — how often the draft’s token survives verification — and k the number of proposals per round. The expected number of tokens emitted per target forward pass is:

E[tokens per target step] = (1 − α^(k+1)) / (1 − α)

Take a well-matched draft with α = 0.8 and k = 4:

E = (1 − 0.8^5) / (1 − 0.8)
  = (1 − 0.32768) / 0.2
  = 0.67232 / 0.2
  ≈ 3.36 tokens per big-model forward pass

So instead of 1 token per expensive target step, you get ~3.36. Since target steps dominate latency, that is roughly a 3.4× cut in the number of sequential big-model passes. Net wall-clock speedup is a bit less — you pay for the draft model’s steps and the slightly heavier verification — but real systems routinely see 2–3× lower latency on the same hardware.

The win scales with α. A draft that mirrors the target well (high α) accepts long runs; a poorly matched draft wastes its proposals:

Acceptance α (k=4)	E[tokens / target step]
0.5	~1.94
0.7	~2.78
0.8	~3.36
0.9	~4.10

This is why the draft must be both cheap and aligned with the target — often a distilled version of it, or the target’s own early layers.

The trade

Speculative decoding spends extra compute and complexity to buy latency:

• You run a second model (more memory for its weights, more engineering) and do verification work that is sometimes thrown away on rejection.
• At very high batch sizes the GPU is already compute-saturated by batching, so the idle-compute headroom shrinks and speculative decoding helps less — it shines most at low batch / latency-sensitive serving, exactly where a single user is waiting on tokens.
• Picking k is a tuning knob: too small wastes the parallel-verify advantage; too large wastes draft work when an early rejection discards the tail.

In dollars-and-latency terms: you convert otherwise-wasted compute into fewer HBM weight-reads per token, trading some peak throughput for a markedly snappier per-request experience. Variants like Medusa (multiple prediction heads) and EAGLE refine the draft step, but the core bargain — guess in bulk, verify in one shot, stay lossless — is unchanged.

It actually happened — two teams land speculative decoding at once (2022–2023)

Speculative decoding arrived in near-duplicate from two groups within months of each other: Google (Leviathan, Kalman, Matias — “Fast Inference from Transformers via Speculative Decoding,” ICML 2023) and DeepMind (Chen et al. — “Accelerating Large Language Model Decoding with Speculative Sampling,” 2023). Both independently derived the same lossless accept/reject rule and reported 2–3× wall-clock speedups on large models with no quality change. The convergence is telling: once you internalize that decode is memory-bound and a verification pass is nearly free, the technique is almost forced. The later Medusa (extra prediction heads on the target itself, so there is no separate draft model) and EAGLE (drafting in feature space) refine how the proposals are made, but every variant rides the same bargain this page describes — guess in bulk, verify in one shot, stay exact.

The architect’s lens

Speculative decoding turns decode’s wasted compute into latency savings — the five questions:

• Why does it exist? Because decode is memory-bound: each target-model step reads gigabytes of weights to emit one token, leaving compute idle — and that idle compute can verify several candidate tokens in one pass almost for free.
• What problem does it solve? The latency of too many sequential big-model steps: a cheap draft model proposes k tokens, the target verifies them all in one prefill-like pass, yielding ~3.36 tokens per target step at α=0.8, k=4 — a ~3.4× cut in sequential passes and 2–3× lower wall-clock latency, losslessly.
• What are the trade-offs? Extra compute and complexity: you run a second model (its weights, its engineering) and sometimes discard verification work on rejection; the draft must be both cheap and aligned (high acceptance α) or it wastes proposals.
• When should I avoid it? At high batch sizes the GPU is already compute-saturated by batching, so the idle-compute headroom vanishes and it barely helps — it shines at low-batch, latency-sensitive serving; skip it if you can’t get a well-matched draft.
• What breaks if I remove it? Decode returns to one token per expensive weight-read — per-request latency rises 2–3× and the idle compute stays wasted — though aggregate throughput at high batch is unaffected, since the technique traded peak throughput for latency anyway.

Check your understanding

1. What property of decode makes verifying k tokens in one pass almost as cheap as generating one?
2. Why is speculative decoding described as lossless, and what mechanism guarantees it?
3. With α = 0.7 and k = 4, compute the expected tokens emitted per target forward pass.
4. What two qualities must the draft model have, and why do both matter for the speedup?
5. Why does speculative decoding help less at very high batch sizes, and what serving regime does it favor?

Show answers

1. Decode is memory-bandwidth-bound: a target step’s cost is dominated by reading the weights from HBM, and that single read can score k candidate tokens in parallel (a prefill-like pass) almost as cheaply as scoring one, since the extra tokens use otherwise-idle compute.
2. Because the accept/resample rule (rejection sampling against the target’s own distribution) makes the emitted tokens have exactly the same distribution as if the target had generated them alone. Wrong guesses are discarded and the token is resampled from a corrected distribution, so quality is identical.
3. E = (1 − 0.7^5) / (1 − 0.7) = (1 − 0.16807) / 0.3 ≈ 0.83193 / 0.3 ≈ 2.77 tokens per target step.
4. It must be cheap (small, so its sequential steps add little latency) and well-aligned with the target (high acceptance rate α, so long runs of proposals survive verification). Low cost without alignment wastes proposals; high alignment from an expensive draft erodes the latency win.
5. At high batch sizes the GPU is already compute-saturated by batching, so there is little idle compute headroom for the “free” parallel verification to exploit. It favors low-batch, latency-sensitive serving where a single user waits on each token.