Prefill vs Decode: The Two Phases

You now have the parts: tokens, attention, and the FFN. This page assembles them into the single most important operational fact about LLM inference: a request runs in two phases with opposite bottlenecks. Internalize this and the rest of the serving world stops being a list of tricks and becomes a set of consequences.

The two phases

When you send a prompt and get a response, the model does two distinct things:

1. Prefill — read the entire prompt at once and build up internal state. Produces the first output token.
2. Decode — generate the rest of the output one token at a time, each new token fed back in as input (autoregression).

  prompt: "Summarize the following report: ...."   (say 2,000 tokens)

  ── PREFILL ────────────────────────┐   ── DECODE ─────────────────────────────
  process all 2,000 tokens together  │   t1 ─► t2 ─► t3 ─► t4 ─► ... ─► tN
  in ONE big parallel pass           │   one token per step, sequentially
  ▲                                  │   ▲                  ▲
  └─ gives TTFT (time to 1st token)  │   └─ each step adds   └─ gives TPOT
                                     │      one TPOT             (time per output token)

The asymmetry is the whole story. Prefill processes thousands of tokens in one shot. Decode then does N tiny sequential steps, one per generated token, and cannot be parallelized across its own time steps — token t+1 depends on token t.

Why prefill is compute-bound

In prefill, all prompt tokens go through the model together. The matmuls are big: a [2000 × d] activation matrix times the weight matrices. This means each weight you load from memory is reused across all 2,000 tokens — high arithmetic intensity (many FLOPs per byte read). That keeps the GPU’s compute units busy, putting prefill on the compute-bound side of the roofline. Cost-wise, prefill is efficient per token: you amortize the expensive weight-loading across the whole prompt in one pass. This is why TTFT for a long prompt is fast relative to its size.

Why decode is memory-bound

In decode, you process one token at a time. To compute that single token, you must still read every weight in the model from memory — all 16 GB for an 8B model — but you only do a sliver of math with them (a [1 × d] vector, not [2000 × d]). Arithmetic intensity collapses: tons of bytes read, few FLOPs done. The GPU’s compute units sit mostly idle while it waits on memory bandwidth. Decode is squarely memory-bound — the conclusion from Part 1, now located in a specific phase.

The opposite-bottleneck insight

Prefill is compute-bound. Decode is memory-bound. Same model, same weights, opposite constraint — purely because of batch shape: prefill works on many tokens at once, decode on one. Almost every serving optimization is a way to make decode look more like prefill (process more tokens per weight-load) or to keep the two phases from starving each other.

The two metrics this produces

Metric	Set by	Means	The user feels it as
TTFT (time to first token)	Prefill	how long until output starts	the “spinner” delay
TPOT (time per output token)	Decode	how fast tokens stream after that	the “typing speed”

Total latency ≈ TTFT + (output_tokens × TPOT). A 2,000-token prompt generating 200 tokens spends one prefill (TTFT) plus 200 sequential decode steps (200 × TPOT). For long generations, decode dominates wall-clock time — even though each step does almost no math — because there are so many sequential steps, each gated on reading the full weights.

Worked example: where does the time go?

Take an 8B model on one GPU. Suppose prefill of a 2,000-token prompt takes 200 ms (TTFT), and each decode step takes 20 ms (TPOT). Generate 300 tokens:

• Prefill: 200 ms, processing 2,000 tokens → 0.1 ms/token.
• Decode: 300 × 20 ms = 6,000 ms, processing 300 tokens → 20 ms/token.

Decode is ~200× slower per token than prefill, and accounts for 6,000 of the 6,200 ms total (97%). The dollars follow the same split: prefill rents the GPU briefly at high utilization; decode rents it for seconds at low utilization. That low utilization is the inefficiency the next parts attack.

Why this dichotomy underlies everything later

Because the two phases have opposite bottlenecks, you optimize them differently — and that single fact seeds much of the book:

• Continuous batching (Part 3): pack many requests’ decode steps together so one weight-load serves many tokens — turning memory-bound decode toward compute-bound.
• Chunked prefill (Part 9): break a giant prefill into pieces so it doesn’t block other requests’ decode steps from getting GPU time.
• Disaggregation (Part 9): run prefill and decode on separate machines tuned for their opposite bottlenecks, so neither starves the other.

And the reason decode can be O(n)-cheap per step at all — rather than recomputing everything — is the subject of the next page.

Under the hood — batched decode is a tiny prefill

Why does continuous batching “turn decode toward compute-bound”? Because the only thing separating the two phases is the shape of the activation matrix, and batching changes that shape. A single request’s decode step multiplies a [1 × d] vector by each weight matrix — one row, so every weight byte you load does just one row’s worth of math. Stack B requests’ decode steps into the same kernel and you are now multiplying a [B × d] matrix by those same weights: one load of the weights, B rows of math.

  decode, batch 1:    [1 × d] · W    → load all weights, do 1 row of work     (memory-bound)
  decode, batch 64:   [64 × d] · W   → load all weights ONCE, do 64 rows      (climbing toward compute-bound)
  prefill, 2000 tok:  [2000 × d] · W → same kernel shape as a big batch        (compute-bound)

A batched decode step is arithmetically the same operation as a prefill of B tokens, which is why it inherits prefill’s better arithmetic intensity. The crossover — where decode stops being starved on bandwidth and starts saturating compute — sits at a batch size on the order of a few hundred on current datacenter GPUs, tracking the chip’s compute-to-bandwidth ratio. Below it you waste compute; above it you waste nothing but begin adding latency. That single picture is the seed of the whole next part.

Check your understanding

1. What does prefill produce, and what does decode produce, in terms of tokens?
2. Why is prefill compute-bound while decode is memory-bound, when both run the same model on the same weights?
3. Define TTFT and TPOT, and say which phase sets each.
4. In the worked example, decode was ~200× slower per token yet that’s expected — why is per-token decode so slow despite doing little math?
5. Name one serving technique that exists because prefill and decode have opposite bottlenecks, and say what it does.

Show answers

1. Prefill processes the whole prompt in one parallel pass and produces the first output token; decode generates the remaining output one token at a time autoregressively.
2. Prefill processes many tokens together, so each weight loaded from memory is reused across all of them (high arithmetic intensity → compute-bound). Decode processes one token at a time, so it must read all the weights but does very little math with them (low arithmetic intensity → memory-bound).
3. TTFT (time to first token) is the latency to produce the first token, set by prefill. TPOT (time per output token) is the streaming speed of subsequent tokens, set by decode.
4. Each decode step must read every weight in the model from memory to produce just one token, and the steps are sequential (each depends on the last), so it’s gated on memory bandwidth, not compute.
5. Examples: continuous batching (pack many requests’ decode steps so one weight-load serves many tokens), chunked prefill (split big prefills so they don’t block decode), or disaggregation (run prefill and decode on separate machines tuned for each bottleneck).