Why LLM Inference Is Memory-Bound

The roofline page ended with a cliffhanger: a matrix-vector product has arithmetic intensity around 1 FLOP/byte, hundreds of times below the ridge point, so it is hopelessly memory-bound. This page cashes that in. It explains why generating text with an LLM is a stack of matrix-vector products, why that gives token latency a hard floor set by memory bandwidth, and why batching is the single most important trick for making it cheaper. This is the most important page in the part.

Prefill vs decode: two very different regimes

Running an LLM has two phases, and they sit on opposite sides of the roofline.

• Prefill processes your prompt. All the prompt’s tokens go through the model at once — say 1000 tokens together. That makes the per-layer work a matrix (1000 tokens) times a matrix (the weights): a real matmul, lots of reuse, compute-bound. Prefill is fast per token.
• Decode generates the answer, one token at a time. Each new token depends on the previous one, so you cannot batch them — token N+1 needs token N to exist first. Each step pushes a single token’s vector through the weights. Matrix times vector. Intensity ~1. Memory-bound.

PREFILL (parallel)              DECODE (sequential)
 [t1 t2 t3 ... t1000]            t1 → t2 → t3 → t4 → ...
   one big matmul                one matrix-vector per token,
   reuse weights across          each weight read again,
   1000 tokens → compute-bound   used once → memory-bound

Almost all the time a user waits for a long answer is spent in decode. So the economics of inference are the economics of one memory-bound matrix-vector step, repeated.

Why one-token-at-a-time is the curse

In decode, every layer multiplies the weight matrices by a vector of size “hidden dimension.” To produce the next token you must read every weight in the model exactly once from HBM, and use each one for a single multiply-add. There is no reuse to amortize the read. The cores are starved; the bottleneck is purely how fast can we stream all the weights out of HBM.

That collapses the latency model to something you can compute on a napkin:

time per token  ≈  bytes read from HBM  /  HBM bandwidth

The bytes read are dominated by the model weights (plus the KV cache, which we get to below). FLOPs barely enter the equation — the chip is waiting, not computing.

Worked example: the latency floor

Take a 70B-parameter model in FP16 (2 bytes/param), served on an H100 (~3.35 TB/s).

weight bytes = 70 × 10^9 params × 2 bytes = 140 × 10^9 bytes = 140 GB

time/token (floor) = bytes / bandwidth
                   = 140 × 10^9 bytes  /  3.35 × 10^12 bytes/s
                   ≈ 0.0418 s
                   ≈ 42 ms per token

So even on perfect hardware with zero overhead, this model cannot emit tokens faster than ~42 ms each ≈ ~24 tokens/second for a single request — and that is a floor, real systems are slower. Notice what did not appear: the H100’s ~1000 TFLOP/s of compute. We never got near it. The model’s intelligence is gated by a memory bus.

The KV cache adds to the bill

Decode also reads the KV cache — the stored keys and values of every prior token — each step. For long contexts this can rival or exceed the weight bytes, raising bytes-read and pushing time/token above the weight-only floor. It is more memory traffic, so it makes the same memory-bound problem worse. The fix is the same family of tricks: store it in fewer bytes (quantized KV) or read less of it.

Why batching is the escape hatch

Here is the beautiful part. To generate one token for one user you read all 140 GB of weights. To generate one token for 32 users at once, you… still read all 140 GB of weights — once — and reuse each weight across all 32 requests. The matrix-vector product becomes a matrix-matrix product again. You climbed back up the roofline.

batch = 1:   read 140 GB → 1 token   → 140 GB per token
batch = 32:  read 140 GB → 32 tokens → ~4.4 GB per token
             (same weight read, amortized 32 ways)

per-token weight bytes ≈ 140 GB / batch_size

The weight read is a fixed cost per step; batching amortizes it across many requests. That is why every serious inference server batches aggressively: it barely changes any single user’s latency (you still read the weights once per step) but multiplies total throughput, which divides the dollar cost per token by the batch size. A GPU-hour costs the same whether it serves 1 user or 32; batching is how you stop paying for an idle, starving chip.

Batch size	Weight bytes/step	Tokens/step	GB per token	Relative $/token
1	140 GB	1	140	1.0×
8	140 GB	8	17.5	~0.13×
32	140 GB	32	4.4	~0.03×

Note

Batching has limits: bigger batches need more HBM for activations and KV cache, and eventually you saturate compute or run out of memory and the gains flatten. But from batch 1 upward, the early gains are enormous — going from 1 to 8 cuts cost-per-token by roughly 8×. This single fact drives the design of every LLM serving stack in the book.

The throughline

Decode is memory-bound, so the lever for making inference cheaper is almost always move fewer weight-bytes per token: shrink the weights (lower precision), or reuse each weight across more requests (batching, and later, speculative decoding). Buying more FLOP/s does nothing for a starving chip. Remember the napkin formula — time/token ≈ bytes_read / bandwidth — and you can predict latency and cost before writing a line of code.

Under the hood — the critical batch size

Batching climbs the roofline, but not forever. Each request you add to the batch reuses every weight one more time before it is discarded, so decode’s arithmetic intensity rises roughly in step with batch size: batch 1 is about 1 FLOP/byte, batch 8 about 8, and so on. The roofline page pinned the ridge point for FP16 on an H100 at ~300 FLOPs/byte — and that number is also a batch size. Below a batch of a few hundred you are memory-bound and each extra request is nearly free throughput; once the batch pushes intensity past the ridge, decode flips to compute-bound and further batching mostly trades latency for very little extra throughput.

That crossover is the critical batch size, and it explains the shape of the table above: the jump from batch 1 to 8 to 32 buys enormous savings because you are deep in the memory-bound region where the GPU was starving. The savings flatten as you approach the ridge, because by then the compute units are finally the thing you are fully using. In practice you rarely reach it for big models — KV-cache and activation memory run out first — but knowing it exists tells you when to stop turning the batching knob and reach for a different lever instead.

Check your understanding

1. Why is prefill compute-bound while decode is memory-bound, even though both run the same model?
2. Write the napkin formula for time-per-token in decode and explain why FLOP/s is absent.
3. Compute the per-token latency floor for a 140 GB model on a GPU with ~2.0 TB/s bandwidth (an A100).
4. Mechanically, why does batching 32 requests together not increase the weight bytes read per step?
5. Why does batching slash dollar-cost-per-token even though it barely improves a single user’s latency?
6. Why don’t the savings from batching continue indefinitely — what is the “critical batch size”?

Show answers

1. Prefill processes all prompt tokens together, so each weight is reused across many tokens (a matmul, high intensity → compute-bound). Decode generates one token at a time and depends on the prior token, so each step is a matrix-vector product reusing each weight once (intensity ~1 → memory-bound).
2. time/token ≈ bytes_read / bandwidth. FLOP/s is absent because the chip is waiting on HBM to deliver weights, not computing; the compute units are mostly idle.
3. 140 × 10^9 bytes ÷ 2.0 × 10^12 bytes/s ≈ 0.070 s ≈ 70 ms per token (~14 tokens/s).
4. The weights are read once per step regardless of batch; the 32 requests share that single read, each reusing the same weights — it turns the matrix-vector into a matrix-matrix product without re-reading weights.
5. Each step still reads the weights once (so per-user latency is roughly unchanged), but that fixed weight-read is amortized across all batched requests, so total tokens produced per GPU-second — and thus tokens per dollar — rises with batch size.
6. Batching raises decode’s arithmetic intensity roughly in proportion to batch size; once that intensity reaches the roofline’s ridge point (~300 FLOPs/byte for FP16 on an H100), decode becomes compute-bound. That crossover is the critical batch size — beyond it the compute units are saturated, so more batching adds latency and memory pressure while buying little extra throughput.