Chunked Prefill

Scheduling and queueing is about deciding which request runs next; the two phases page is about why prefill and decode fight for the same GPU. Chunked prefill is the scheduler’s answer to that fight, done on a single shared GPU instead of separate pools.

The problem: one big prefill blocks everyone

Picture a GPU happily streaming tokens to 30 users — a steady drumbeat of decode steps, one per user every ~20 ms. Then a request arrives with a 16,000-token prompt. Its prefill is one enormous compute job. A naive scheduler runs it to completion before resuming anyone else.

  decode: d d d d │■■■■■■■■■■■■■■■■■■■■■■■■│ d d d d
                  │   16K-token PREFILL    │
                  └── ~300 ms, all 30 users' streams frozen ──┘

The damage, in the metrics that matter:

• The big request’s TTFT is fine — it monopolized the GPU.
• Every other user’s TPOT spikes by the whole prefill duration. Their tokens stop for ~300 ms. On a streaming UI that is a visible, ugly stall — and your p99 TPOT just fell off a cliff.

This is head-of-line blocking: one fat job at the front starves a crowd of small ones behind it.

The fix: chop the prefill, interleave with decode

Chunked prefill (from the Sarathi/Sarathi-Serve work, now standard in vLLM) splits a large prefill into fixed-size chunks — say 512 tokens each — and runs them one chunk per scheduler step, interleaved with the ongoing decode steps of all the other users.

  step:  [P₁ + d×30] [P₂ + d×30] [P₃ + d×30] ... [P₃₂ + d×30]
          512 tok      512 tok      512 tok         512 tok
          chunk 1      chunk 2      chunk 3         chunk 32
          ▲ each step also carries everyone's decode tokens ▲

Instead of one 300 ms blackout, the prefill becomes 32 small slices, and between every slice the 30 decode streams get a turn. No user waits more than one chunk-time (a few ms) for their next token. The big request’s TTFT grows slightly — it now finishes over 32 steps instead of one — but the fleet’s tail latency stays flat.

The bonus: piggyback decode fills idle compute

There is a second, subtler win, and it comes straight from the opposite-bottleneck insight.

• A pure decode batch is memory-bound — the compute units sit idle while weights stream in.
• A prefill chunk is compute-bound — it has spare memory bandwidth.

Mix them in the same batch and they cover each other’s weaknesses: the prefill chunk uses the idle compute, the decode tokens ride along on weight-loads that were happening anyway. You raise GPU utilization (MFU) without raising the bill — the decode tokens are effectively free, piggybacked onto a step the prefill was paying for.

The core trade

Chunked prefill converts a large, rare, catastrophic latency spike into a small, constant, tolerable overhead spread across every step. You give up a little TTFT on big prompts and a little scheduling overhead; you get smooth p99 TPOT and a fuller GPU. For most production SLAs that’s a great trade.

Worked example: tail latency, before and after

GPU serving 30 users; a 16K-token prefill that would take 320 ms as one job; chunk size 512 tokens → 32 chunks of ~10 ms each. Decode step ~20 ms.

• Naive: during the prefill, every other user’s next token is delayed by up to 320 ms. Worst-case added TPOT ≈ 320 ms.
• Chunked: between each ~10 ms chunk the decode batch runs, so any user waits at most ~one chunk before their token. Worst-case added TPOT ≈ ~10–30 ms.

Same total prefill work, same first-token deadline roughly met — but the p99 TPOT inflicted on bystanders drops by ~10×. The cost is a modest increase in the big request’s TTFT (it spans 32 steps) and slightly more scheduler bookkeeping per step.

Tuning the chunk size

Chunks too large → you’re back toward blocking. Too small → per-chunk overhead dominates and prefill throughput drops (you stop amortizing weight-loads). The sweet spot balances bystander TPOT against the big request’s TTFT and overall throughput; it depends on model, hardware, and your SLA.

Chunked prefill vs disaggregation

Both solve prefill–decode interference; they sit at different scales.

	Chunked prefill	Disaggregation
Hardware	one shared GPU	separate GPU pools
Removes interference?	softens it	eliminates it
Extra cost	scheduling overhead	KV-cache transfer
Best for	most deployments	very large fleets

They also compose: chunk within each pool and disaggregate the pools.

Under the hood — the per-step token budget

The actual scheduling primitive is a token budget per iteration. Each step the engine may process some fixed number of tokens — vLLM exposes this as max_num_batched_tokens. It fills that budget decode-first: one token for every running request (those are latency-critical and must not wait), and whatever budget is left over is handed to a slice of a waiting prefill. A 16K-token prompt with, say, 480 tokens of budget remaining after the decodes simply takes ~34 steps to ingest. Sarathi-Serve calls the result stall-free batching — no step is ever a pure prefill that freezes the decoders, because decode always has first claim on the budget. Tuning “chunk size” is really tuning this one number: a bigger budget means fatter prefill chunks (better prefill throughput, riskier TPOT); a smaller one means smoother streams at the cost of prefill efficiency.

The cost lens

Chunked prefill spends almost nothing — a touch of overhead and a touch of TTFT — to buy a smooth token stream and higher utilization. It is the cheapest fix for the most common scaling complaint (“the model keeps stuttering when someone pastes a huge prompt”), which is why it became a default rather than an exotic option.

The architect’s lens

A near-free scheduler tweak that trades a catastrophe for a constant:

• Why does it exist? Because prefill is compute-bound and decode is memory-bound, so one fat 16K-token prefill run-to-completion freezes every other user’s token stream — head-of-line blocking — and tanks p99 TPOT.
• What problem does it solve? Bystander tail latency: slicing the prefill into ~512-token chunks interleaved one-per-step with everyone’s decode bounds the worst-case added TPOT from ~320 ms to ~10–30 ms (a ~10× cut), and piggybacked decode fills the prefill chunk’s idle compute for free.
• What are the trade-offs? The big request’s TTFT grows slightly (its prefill now spans ~32 steps) plus a little per-step scheduling overhead — you tune one number (max_num_batched_tokens) to balance bystander TPOT against prefill throughput.
• When should I avoid it? Chunks too large reintroduce blocking; too small and per-chunk overhead dominates and prefill throughput drops — and at very large fleet scale disaggregation eliminates the interference rather than softening it.
• What breaks if I remove it? A naive scheduler runs each big prefill to completion, so the fleet’s token streams stutter every time someone pastes a huge prompt and p99 TPOT falls off a cliff.

Check your understanding

1. In the naive case, whose latency suffers when a 16K-token prompt arrives, and which metric specifically?
2. What does chunked prefill do differently, and how does that bound how long any user waits for their next token?
3. Why does mixing a prefill chunk with decode tokens in one batch raise GPU utilization “for free”?
4. What does the big request give up under chunked prefill, and why is that usually an acceptable trade?
5. How does chunk size affect the trade-off, and what goes wrong at each extreme?

Show answers

1. The big request’s TTFT is fine, but every other user’s TPOT spikes — their token stream freezes for the whole prefill duration (head-of-line blocking).
2. It splits the prefill into fixed-size chunks and runs one chunk per scheduler step interleaved with everyone’s decode steps, so any user waits at most ~one chunk-time (a few ms) for their next token instead of the whole prefill.
3. Decode is memory-bound (idle compute), prefill is compute-bound (spare bandwidth); mixed in one batch the prefill uses the idle compute while decode tokens ride existing weight-loads, so the decode tokens are effectively free.
4. The big request’s TTFT increases slightly because its prefill now spans many steps; that’s acceptable because it keeps fleet-wide p99 TPOT flat and improves utilization.
5. Chunks too large reintroduce blocking; chunks too small make per-chunk overhead dominate and hurt prefill throughput (weight-loads stop being amortized). The optimum balances bystander TPOT, the big request’s TTFT, and throughput.