Training vs Inference: Two Cost Regimes

What a Model Actually Is gave us params, FLOPs, and bytes. Those same numbers behave very differently depending on what you’re doing with the model. “Running a model” splits into two regimes with opposite cost profiles: training and inference. Confusing them is the most common source of muddled cost reasoning, so we separate them cleanly here — and explain why this book lives mostly in the inference camp.

Training: forward + backward + optimizer

Training teaches the model by showing it data and adjusting the weights. Each step has three phases:

1. Forward pass — same as inference: run the input through, get a prediction.
2. Backward pass — compute gradients (how to nudge every weight to reduce error). This is roughly twice the FLOPs of the forward pass.
3. Optimizer step — update all the weights using those gradients.

The memory blows up because training must keep the activations from the forward pass (needed to compute gradients) and store optimizer state. A common rule: training a model in mixed precision needs roughly 16–20 bytes per parameter of memory — weights, gradients, and optimizer moments — versus the ~2 bytes/param for FP16 inference. That’s why a 7B model you can serve on one GPU may need a small cluster to train.

Per training step, per token:
  forward   ~2 × params FLOPs
  backward  ~4 × params FLOPs
  total     ~6 × params FLOPs   (≈ 3× the inference cost per token)

Crucially, training is throughput-bound: you process enormous batches, nobody is waiting on any single example, and you optimize for examples per second. Latency per example is irrelevant.

Inference: forward only, but billions of times

Inference is just the forward pass — no backward, no optimizer. Per token it’s cheaper than training (~2× params vs ~6× params FLOPs) and needs far less memory (~2 bytes/param for weights). So why is it the focus of this book?

Because of how often you do it. A model is trained essentially once (plus occasional fine-tunes), but then serves billions of requests over its lifetime. The lifetime inference bill dwarfs the training bill for any successful product. And inference is latency-sensitive: a real user is waiting for the first token and for each token after it, so you cannot hide cost behind giant batches the way training does. You must balance batching (for throughput) against responsiveness (for latency) — a tension that drives most of Parts 3 and 5.

There’s a subtler twist. Token generation (decode) reuses the whole model to emit one token at a time, doing very little arithmetic per byte of weights it must read. That makes decode memory-bandwidth-bound, not compute-bound — the GPU spends its time fetching weights, not multiplying. Training, with its huge batches, keeps the compute units busy and tends to be compute-bound. Same hardware, opposite bottleneck. We make this precise in Why LLM Inference Is Memory-Bound.

Side by side

Dimension	Training	Inference
Passes	Forward + backward + optimizer	Forward only
FLOPs/token	~6 × params	~2 × params
Memory/param	~16–20 bytes (weights, grads, optimizer, activations)	~2 bytes (weights) + KV cache
How often	Once (+ fine-tunes)	Billions of times
Optimize for	Throughput (examples/sec)	Latency and throughput together
Typical bottleneck	Compute-bound	Memory-bandwidth-bound (decode)
Who waits	Nobody	A user, right now
Lifetime $	Big one-time capital cost	Ongoing, usually the larger total

Why this book centers inference

If you operate AI systems, the recurring monthly bill — the thing that scales with your users and can sink your margins — is inference. Training is a periodic capital expense handled by a few teams; inference is the operational cost everyone pays, forever, on every request. So the spine of this book (Parts 1–5) is inference and serving, and we devote Part 7 to training as the important-but-secondary regime. When you reach for an optimization, always ask first: am I in the train-once world or the serve-forever world? The cheapest lever is almost always the one on the side you pay most often.

The cost asymmetry in one line

Training cost ≈ (model size) paid once. Inference cost ≈ (model size) × (every token of every request, forever). Multiply the second by your traffic and you’ll see why we obsess over the token.

Under the hood — gradient checkpointing trades compute back for memory

The ~16–20 bytes/param figure assumed you keep every forward-pass activation around until the backward pass needs it. For a deep model with long sequences, those saved activations can dwarf even the weights and optimizer state — and they scale with batch size and sequence length, not just parameter count. So training frameworks reach for a classic trade: don’t store most activations; recompute them.

This is gradient checkpointing (also called activation recomputation). You keep activations only at a handful of layer boundaries (“checkpoints”) and discard the rest. When the backward pass needs an activation you didn’t save, you redo the small slice of forward work to regenerate it. The result is a deliberate move through the same trade-off space this book keeps returning to: you spend extra compute (often roughly 30% more forward FLOPs) to buy back a large chunk of activation memory, which lets you fit a bigger model or a bigger batch on the same GPUs.

It is the training-side mirror of the inference tricks ahead. Decode saves memory bandwidth by reading fewer weight-bytes; training saves memory by recomputing activations instead of storing them. Same mindset — find the resource you are short on, and pay in the one you can spare.

Check your understanding

1. What three phases make up a training step, and roughly how do their FLOPs compare to the forward pass?
2. Why does training need ~16–20 bytes per parameter while FP16 inference needs only ~2?
3. Inference is cheaper per token than training. Why, then, is inference usually the larger lifetime cost?
4. Explain why decode is typically memory-bandwidth-bound while training is compute-bound.
5. Give the practical reason this book makes inference its spine and treats training as secondary.
6. What does gradient checkpointing trade, and why would a training team accept that trade?

Show answers

1. Forward pass (~2 × params FLOPs), backward pass (~4 × params FLOPs), and the optimizer step. Total ~6 × params FLOPs per token, roughly 3× the inference forward pass.
2. Training must store weights plus gradients, optimizer state (e.g. momentum/variance), and the saved activations needed to compute gradients; inference only needs the weights (plus a runtime KV cache).
3. A model is trained once but then serves billions of requests; multiplied across all that traffic over the model’s lifetime, the per-token cost adds up to far more than the one-time training cost.
4. Decode generates one token at a time, reusing the entire model with very little arithmetic per byte of weights read — so the GPU is bottlenecked fetching weights from memory. Training processes huge batches that keep the compute units saturated, so it’s limited by arithmetic throughput.
5. Inference is the recurring operational cost that scales with users and traffic — the bill that determines product margins — whereas training is a periodic, one-time capital expense handled by few teams.
6. It throws away most forward-pass activations and recomputes them during the backward pass, spending extra forward FLOPs (often ~30% more) to drastically cut activation memory — worthwhile when activation memory, not compute, is what blocks a larger model or batch from fitting on the GPUs.