The 2025 Serving Frontier

Disaggregated prefill and decode was, when this book introduced it, a “large-fleet” optimization — clever, but rare. The story of serving as of ~2025 is that the techniques in the Advanced part stopped being exotic and started becoming defaults. This page surveys four shifts that, taken together, define the modern serving stack — and each one is the same throughline answer made structural: what does this cost in latency, memory, and dollars, and how do we make it cheaper? Treat system names as representative examples of an idea, not as the final word; the field moves fast.

Four shifts, one table

  Was "advanced" (rare)            Becoming default (~2025)
  ─────────────────────            ────────────────────────
  prefill/decode disaggregation →  mainstream in large fleets
  prefix caching (per request)  →  prefix sharing across users
  KV cache lives in GPU HBM     →  KV tiered GPU → CPU → SSD
  speculative decoding (opt-in) →  a standard latency lever

None of these is a new invention. What changed is that they moved from research papers and frontier labs into the serving frameworks ordinary teams deploy.

Disaggregation goes mainstream

The interference problem — a bursty, compute-bound prefill stalling everyone’s memory-bound decode stream — gets worse as fleets grow and prompts get longer. The fix, splitting prefill and decode onto separate GPU pools tuned to their opposite bottlenecks, was proven in production by two 2024 systems the earlier page already named: DistServe (OSDI 2024) made the goodput case quantitatively, and Mooncake (the architecture behind Moonshot AI’s Kimi) productionized a KV-cache-centric version.

By ~2025 the idea is no longer the preserve of a few labs — disaggregation patterns are showing up as first-class options in mainstream serving frameworks. The cost it trades is unchanged: you pay a KV-cache transfer between pools (cheap over NVLink/InfiniBand, real over slower cross-node links) in exchange for converting unpredictable, interference-driven tail latency into a tunable, transferable fee.

Prefix sharing across requests and users

Prefix caching reused a shared prompt’s KV across one application’s requests. The frontier move is to make that cache a cluster-level, cross-user resource: any decode worker can reuse a hot prefix that any request — possibly from a different user — already computed.

  Per-request caching:  user A's requests reuse A's system-prompt KV
  Cross-user sharing:   a hot prefix computed once is reused by
                        every worker, across users, fleet-wide

This is the natural endpoint of Mooncake’s “disaggregated KV cache” idea: once the KV cache is network-addressable rather than pinned to one GPU, a popular prefix (a shared system prompt, a common few-shot block, a frequently-retrieved document) is prefilled once for the whole cluster. The win is the bluntest in the book — the cheapest token is the one you never compute — scaled from one process to the entire fleet.

Sharing KV across users needs care

Cross-user prefix reuse is exact and lossless for the shared tokens (causality guarantees identical KV for an identical prefix). But it raises real concerns the per-request version didn’t: you must ensure only genuinely public/shared prefixes are pooled, that nothing private leaks via cache-hit timing, and that eviction policy is fair. The mechanism is correct by construction; the policy around what may be shared is where the engineering lives.

KV-cache offloading and tiering

At long context the KV cache can outgrow the weights themselves, and GPU HBM is the scarcest, most expensive memory in the system. The frontier answer is to stop treating HBM as the only home for KV and build a memory hierarchy for it, exactly as CPUs have done for decades:

  ┌──────────┐  fastest, smallest, $$$$   ← active KV, hot prefixes
  │ GPU HBM  │
  ├──────────┤  ~10× bigger, cheaper      ← warm KV, recently used
  │ CPU DRAM │
  ├──────────┤  huge, cheap, slow         ← cold prefixes, big cache
  │   SSD    │
  └──────────┘

Cold or paused requests’ KV is offloaded down the tiers and pulled back up on demand. This lets a server hold far more concurrent or paused conversations than HBM alone could fit, and lets a cross-user prefix cache be large enough to actually get hits. The cost is the transfer latency when a tier miss forces a fetch up the hierarchy — so the engineering is the same caching discipline as always: keep the hot set high, accept that the cold set is slow.

By the numbers — why tiering buys so much headroom

The numbers come straight from the KV-cache math. Take a model that spends on the order of ~0.5–1 MB of KV per token. A single 100K-token context is then tens of GB — a large bite out of an 80 GB GPU before you’ve served anyone else.

  100K tokens × ~0.8 MB/token ≈ 80 GB of KV for ONE long context

You cannot hold many such contexts in HBM at once. But CPU DRAM is often ~10× larger and SSD far larger still, so offloading cold contexts down the tiers lets one node retain hundreds of paused conversations (and a big shared-prefix cache) for the price of slower reactivation when one wakes. You trade a one-time fetch latency for a massive increase in how much state a server can keep alive — memory for latency, the book’s recurring trade, applied to the KV cache itself.

Speculative decoding becomes a default

Speculative decoding — a small draft model proposes tokens, the big model verifies them in one parallel pass, losslessly — was a 2022–2023 result that took time to harden. As of ~2025 it’s a standard latency lever, with the draft-quality refinements (Medusa-style extra heads, EAGLE-style feature-space drafting) widely available in serving stacks. The reason it matters more now: as more workloads are latency-sensitive single-user sessions (chat, agents, coding assistants) running at low batch size, the idle-compute headroom speculative decoding exploits is exactly what’s available — so it shines precisely where the modern workload lives.

The shape of the modern stack

Stitch the four together and a picture emerges:

  request ─▶ [ prefill pool ] ──KV──▶ [ decode pool ]
                  │                         │
            check cluster KV cache    speculative decoding
            (cross-user prefix reuse) (draft + verify)
                  │                         │
            KV tiered across GPU HBM → CPU DRAM → SSD

Every box is one of the book’s levers, now wired together as a system rather than deployed in isolation. The unifying principle is the one from the Advanced overview — stop forcing one resource to do incompatible jobs, and stop paying for work you’ve already done — pushed to the fleet level.

It actually happened — Mooncake’s KV cache became infrastructure (2024)

The clearest marker that the KV cache had graduated from “per-GPU detail” to “first-class resource” was Mooncake, the serving architecture behind Moonshot AI’s Kimi assistant, described publicly in 2024. Its design treats the KV cache as a disaggregated, cluster-wide pool spread across the spare CPU DRAM and SSD capacity of the whole cluster — not memory that belongs to any one GPU. A prefix prefilled by one worker becomes reusable by any other, and cold KV lives on cheaper tiers until needed. That single architectural decision — make the KV cache network-addressable and tiered — is the seed of three of the four shifts on this page (disaggregation, cross-user prefix sharing, and KV tiering). It is the canonical, dated example of the 2025 serving frontier taking shape in a real production system.

The cost lens

The 2025 serving frontier is the throughline applied to the fleet rather than the request. Disaggregation makes tail latency tunable instead of random; cross-user prefix sharing erases redundant prefill across the whole cluster; KV tiering lets cheap memory absorb state that expensive HBM can’t hold; speculative decoding converts idle compute into fewer sequential weight-reads. None of it changes the model — it changes how the fleet spends. The direction is unmistakable and singular: drive the cost per useful token down, one structural redundancy at a time.

Check your understanding

1. Name the four serving shifts this page describes, and the “advanced/rare” practice each one replaces.
2. How does cross-user prefix sharing differ from per-request prefix caching, and what new policy concern does it introduce that the per-request version didn’t?
3. Why does treating the KV cache as a GPU HBM → CPU DRAM → SSD hierarchy let a server keep far more conversations alive, and what does it cost?
4. Using the KV-cache math, roughly how much HBM does a single 100K-token context consume at ~0.8 MB/token, and why does that motivate tiering?
5. Why does speculative decoding fit the modern workload especially well as of ~2025?

Show answers

1. Prefill/decode disaggregation (replacing co-located shared-GPU serving), cross-user prefix sharing (replacing per-request-only prefix caching), KV tiering across GPU/CPU/SSD (replacing KV living only in GPU HBM), and speculative decoding as a default (replacing opt-in/absent speculative decoding).
2. Per-request caching reuses a prefix’s KV only within one application’s own requests; cross-user sharing makes the KV a cluster-wide resource any worker can reuse across users. The new concern is policy/safety: ensuring only genuinely public prefixes are pooled, preventing leakage via cache-hit timing, and keeping eviction fair — the mechanism is exact, but what may be shared is a governance question.
3. Cold or paused requests’ KV is offloaded down to larger, cheaper tiers (CPU DRAM, then SSD) and pulled back on demand, so the server holds far more state than scarce HBM allows. The cost is the transfer/fetch latency when a tier miss forces pulling KV back up the hierarchy.
4. 100K × ~0.8 MB ≈ ~80 GB for a single context — roughly a whole large GPU’s HBM for one conversation. You can’t hold many such contexts in HBM, so offloading cold ones to ~10×-larger DRAM and far-larger SSD is what makes many concurrent/paused long contexts affordable.
5. Modern workloads are increasingly latency-sensitive single-user sessions (chat, agents, coding) at low batch size, where the GPU has idle compute headroom. Speculative decoding’s free parallel verification exploits exactly that headroom, so it helps most precisely where today’s traffic lives (and less at high-batch saturation).