Disaggregated Prefill & Decode

The two phases have opposite bottlenecks: prefill is compute-bound, decode is memory-bound. That page warned the two can starve each other. This page is the consequence taken to its logical end — give each phase its own hardware.

The interference problem

On a single GPU running continuous batching, prefill and decode share the same compute units, the same memory bandwidth, and the same scheduler slots. They do not coexist gracefully.

A decode step is tiny and frequent: read all the weights, do a sliver of math, emit one token, repeat every ~20 ms. A prefill is huge and bursty: when a 4,000-token prompt arrives, it saturates the compute units for tens of milliseconds in one shot. While that prefill runs, the decode steps for every other user already mid-response wait their turn.

  Single GPU, shared:
  decode  d d d │■■■■■■■■■■■■■■■■■■│ d d d
                │   PREFILL spike  │
                └── everyone's TPOT stalls here ──┘

The user-visible damage:

• TTFT for the new request is fine — it got the whole GPU.
• TPOT for everyone else spikes — their tokens stop streaming until the prefill clears.

You cannot tune the GPU for both. Prefill wants raw FLOPs and large batches; decode wants memory bandwidth and tiny, latency-critical steps. Optimizing for one pessimizes the other.

Disaggregation: split the pools

Disaggregated serving runs prefill and decode on separate GPU pools. A request flows through both:

  ┌── PREFILL pool ──┐         ┌── DECODE pool ──┐
  │ compute-heavy    │  KV     │ bandwidth-heavy │
  │ big batches      │ ──────► │ many concurrent │
  │ high FLOPs/$     │ cache   │ streams, low    │
  │ produces TTFT    │ over    │ TPOT            │
  └──────────────────┘ NVLink/ └─────────────────┘
                       RDMA

1. Prefill pool ingests the prompt, runs the one big parallel pass, and produces the KV cache plus the first token.
2. The KV cache is transferred over the interconnect to the decode pool.
3. Decode pool generates the rest, one token at a time, batched across many users.

Now each pool is tuned for its own bottleneck. A prefill spike in pool A never touches the decode streams in pool B — they are different silicon. TPOT becomes smooth and predictable; TTFT and TPOT can be scaled independently by adding cards to whichever pool is the constraint. This is how very large deployments (DeepSeek’s serving stack, the Mooncake architecture behind Kimi) keep tail latency flat under bursty traffic.

Why split, not just schedule better

Chunked prefill softens interference on one GPU. Disaggregation removes it by giving the phases different machines. Chunking is cheaper to deploy; disaggregation scales further and lets you right-size each pool. Large fleets often do both.

The cost: moving the KV cache

Nothing is free. The new bill is the KV-cache transfer between pools — and the KV cache is not small.

Worked example. Take a 70B model, 80 layers, hidden split into KV heads giving ~8 KV heads × 128 dims, FP16 (2 bytes), for a 4,000-token prompt. A common KV size is roughly:

KV bytes ≈ 2 (K and V) × layers × kv_dim × bytes × tokens
        ≈ 2 × 80 × 1024 × 2 × 4000
        ≈ 1.3 GB

That ~1.3 GB must cross the interconnect before decode can start. Over NVLink (~100s of GB/s) that is sub-millisecond to a few ms — negligible against a multi-hundred-ms prefill. Over slower RDMA/Ethernet between nodes (tens of GB/s) it can be tens of ms — now a real slice of TTFT.

So disaggregation pays off when:

• The interconnect is fast (NVLink, InfiniBand) relative to the KV size, and
• Traffic is bursty enough that interference was hurting you, and
• You are large enough that dedicating whole pools is affordable.

When NOT to disaggregate

At small scale, two pools means idle capacity — a half-empty prefill pool and a half-empty decode pool cost more than one shared, well-scheduled GPU. The transfer also adds latency and a failure surface. Disaggregation is a large-fleet optimization; below that, chunked prefill on shared GPUs wins on dollars.

It actually happened — DistServe and Mooncake put disaggregation in production (2024)

Two 2024 systems turned this idea from theory into deployed practice. DistServe (Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, Zhong et al., OSDI 2024) made the case quantitatively: by placing prefill and decode on separate pools and tuning each to its own TTFT and TPOT target, it served substantially more requests within SLO — the goodput metric — than a single co-located pool. Mooncake, the architecture behind Moonshot AI’s Kimi assistant, productionized a KV-cache-centric version: not just separate prefill and decode pools, but a disaggregated KV cache spread across the cluster’s spare CPU and SSD memory, so a cached prefix can be reused by any decode worker. The shared lesson: once you split the phases, the KV cache stops being a per-GPU detail and becomes a first-class, network-addressable resource you schedule around.

The payoff in one line

Disaggregation converts an unpredictable, interference-driven tail latency into a tunable, transferable cost: you trade “prefill randomly stalls decode” for “we pay a known KV-transfer fee and size each pool to its bottleneck.” For a fleet whose SLA is TPOT, that trade is often decisively worth it.

The architect’s lens

The logical end of “prefill and decode have opposite bottlenecks” — give each its own silicon:

• Why does it exist? Because on one shared GPU a bursty prefill saturates the compute and stalls everyone’s decode streams (their TPOT spikes), and you can’t tune one GPU for both raw-FLOPs prefill and bandwidth-hungry decode.
• What problem does it solve? It eliminates the interference and lets you scale TTFT and TPOT independently — add cards to whichever pool is the constraint — keeping tail latency flat under bursty traffic (DistServe, Mooncake).
• What are the trade-offs? A new bill: the ~1.3 GB-per-long-prompt KV cache must cross the interconnect — trivial over NVLink (a few ms) but tens of ms over cross-node RDMA/Ethernet, a real slice of TTFT.
• When should I avoid it? At small scale, where two pools leave idle capacity that costs more than one shared, well-scheduled GPU — and when the interconnect is slow relative to KV size; below large-fleet scale, chunked prefill wins on dollars.
• What breaks if I remove it? You’re back to co-located serving where prefill spikes randomly poison decode TPOT — fine for a small deployment, but a TPOT-SLA fleet under bursty load can’t hold its tail.

Check your understanding

1. Why do prefill and decode interfere when they share one GPU, and which metric suffers most for other users during a prefill spike?
2. What exactly gets transferred between the prefill pool and the decode pool, and why is its size the main new cost?
3. How does disaggregation let you scale TTFT and TPOT independently?
4. In the worked example, why is a ~1.3 GB KV transfer trivial over NVLink but potentially costly over cross-node Ethernet?
5. Give two conditions under which disaggregation is not worth it.

Show answers

1. They share compute units, bandwidth, and scheduler slots; a big bursty prefill saturates the GPU and the small frequent decode steps for everyone already mid-response wait — so other users’ TPOT spikes even though the new request’s TTFT is fine.
2. The KV cache (plus first token) is transferred from the prefill pool to the decode pool. It can be ~1 GB+ for a long prompt on a large model, so the interconnect bandwidth becomes the new bottleneck/cost.
3. Each phase runs on its own pool, so you add cards to whichever pool is the constraint — more prefill GPUs to cut TTFT, more decode GPUs to cut TPOT — without affecting the other.
4. NVLink moves 100s of GB/s, so 1.3 GB crosses in a few ms (negligible vs a multi-hundred-ms prefill); cross-node Ethernet at tens of GB/s makes the same transfer tens of ms, a real fraction of TTFT.
5. At small scale (two pools leave idle capacity that costs more than one shared GPU) and when the interconnect is slow relative to the KV size (transfer latency eats the benefit). Bursty-traffic interference also has to actually be hurting you.