Where the Time and Money Actually Go

Training vs Inference put us firmly in the serve-forever world. Now we open up a single inference request and ask: where, precisely, do the milliseconds and the dollars go? You cannot make something cheaper until you know which part of it is expensive. The central discipline of this whole book is one sentence: find the dominant cost, and optimize that.

Anatomy of a request’s latency

From “user hits send” to “last token arrives,” a request passes through four stages:

 ┌────────┐   ┌──────────┐   ┌──────────────────────────┐   ┌─────────┐
 │ QUEUE  │ → │ PREFILL  │ → │ DECODE (token by token)  │ → │ NETWORK │
 │ wait   │   │ read the │   │ generate output, one at  │   │ stream  │
 │ for a  │   │ prompt   │   │ a time, reusing KV cache │   │ back    │
 │ slot   │   │ (1 pass) │   │ ●→●→●→●→●→ ...            │   │         │
 └────────┘   └──────────┘   └──────────────────────────┘   └─────────┘
                  ↑                      ↑
            time to first token    time per output token
            (TTFT) ends here       (TPOT) governs here

Stage	What happens	Scales with	Bottleneck
Queue	Request waits for GPU/batch slot	Load, batch policy	Scheduling
Prefill	Whole prompt processed in one parallel pass	Prompt length	Compute-bound
Decode	Output generated one token at a time	Output length	Memory-bandwidth-bound
Network	Tokens streamed to the client	Bytes, distance	I/O

The two you most directly control by model and prompt choices are prefill and decode, and they have opposite characters.

Compute-bound vs memory-bound

Prefill is compute-bound. The entire prompt is processed at once, so the GPU does a big batched matmul that keeps its compute units busy. Doubling the prompt roughly doubles prefill work, but it’s parallel — this is why API input tokens are cheap.

Decode is memory-bound. To emit each token the GPU must read the entire model’s weights from memory but does only a tiny amount of math with them (one token’s worth). It spends its time waiting on memory bandwidth, not computing. This is the single most important asymmetry in LLM serving, and we prove it on the roofline and in Why LLM Inference Is Memory-Bound. It’s also why prefill and decode are treated as two phases throughout the book.

Worked example: decode dominates long outputs

Suppose a request has a 500-token prompt and asks for a 500-token answer, with these (illustrative) per-token timings on our model:

Prefill: 500 prompt tokens processed in parallel  ≈ 50 ms total
Decode:  500 output tokens × 8 ms each (sequential) ≈ 4000 ms total
Queue + network                                    ≈ 100 ms

Total ≈ 4150 ms  →  decode is ~96% of the latency

Even though prefill and decode handle the same number of tokens, decode costs ~80× more wall-clock time here, because its tokens are produced one at a time while prefill’s were processed together. The lesson is blunt: for any request with a substantial output, decode is the thing to optimize — which is exactly why KV caching, batching, and speculative decoding (Part 3) all target the decode loop. Shave 8 ms → 5 ms per token and you’ve cut nearly the whole request.

Now flip it: a 5,000-token prompt with a 20-token answer (think classification or extraction). Prefill might be ~500 ms and decode only ~160 ms — now prefill dominates, and the right levers are prompt caching and context trimming. The dominant cost is request-shaped, not universal.

Most of the dollars are GPU time

Latency and dollars are tightly linked because the expensive resource is the GPU, billed by time. Storage, the load balancer, logging — rounding error next to GPU-seconds. So “cut the dollars” almost always reduces to one of two moves:

• Make each token take less GPU time (faster decode, lower precision, better kernels).
• Serve more tokens per GPU-second (batching, so one weight-read serves many users at once).

The optimizer’s first question

Before touching anything, profile the request and ask which stage owns the latency and the GPU-time? Optimizing a stage that’s 4% of the bill is wasted effort no matter how clever the fix. Measure, find the dominant cost, and aim there. The rest of this book is a catalog of levers — the skill is pointing the right one at the right stage.

This anatomy is the backbone of everything ahead. Part 1 explains why decode is memory-bound at the hardware level; Parts 3–5 are essentially a toolbox for attacking whichever stage dominates your traffic.

Check your understanding

1. List the four stages a request passes through and what each contributes to latency.
2. Why is prefill compute-bound while decode is memory-bound?
3. In the worked example, why does decode take ~96% of the latency despite handling the same token count as prefill?
4. Describe a request shape where prefill, not decode, is the dominant cost.
5. “Cutting dollars usually reduces to two moves.” What are they, and why is GPU time the thing that matters?

Show answers

1. Queue (waiting for a GPU/batch slot), prefill (processing the whole prompt in one parallel pass — sets TTFT), decode (generating output one token at a time — governs TPOT), and network (streaming tokens back to the client).
2. Prefill processes all prompt tokens in one big parallel matmul that saturates the compute units. Decode emits one token at a time, reading the entire model’s weights from memory but doing little math per read, so it waits on memory bandwidth.
3. Decode’s 500 tokens are produced sequentially (~8 ms each ≈ 4000 ms), while prefill’s 500 tokens were processed together in ~50 ms. Same token count, but one is serial and the other parallel.
4. A long prompt with a short output — e.g. a 5,000-token document with a 20-token classification/extraction answer — makes prefill the dominant cost; levers are prompt caching and context trimming.
5. Make each token take less GPU time (faster/lower-precision decode, better kernels) or serve more tokens per GPU-second (batching). GPU time matters because the GPU is the expensive resource billed by the second; everything else is rounding error.