Part 7 · Training & Fine-Tuning Efficiency

Most of this book lives in the serve-forever world — training vs inference drew that line and explained why inference is the bill that scales with your users. Part 7 crosses to the other side. Training is the train-once regime: a periodic, capital-heavy job that a few teams run, that needs a different mental model because its costs land in different places. We keep this part deliberately short, because the honest truth is that most readers will fine-tune a model, or choose not to train at all — almost nobody pretrains from scratch.

Why training gets its own (small) part

Inference reads the weights and emits tokens. Training reads the weights, computes how to change them, and stores everything needed to do so — gradients, optimizer state, and saved activations. That extra bookkeeping multiplies memory by roughly 8–10× over inference, which is why a 7B model you can serve on a single GPU may need a small cluster to train. The cost throughline still rules every page, but the answers shift: in training, memory is the wall you hit first, dollars come from renting many GPUs for hours or days, and latency-per-example simply does not matter (nobody is waiting on one training example).

The good news: you rarely pay the full bill. The techniques in this part are mostly about not training the whole thing — sharding the state so it fits, or freezing 99% of the weights and training a sliver. That is where the leverage is, and where your money goes furthest.

The roadmap

Four pages, each one a lever on the training bill.

1. Why Training Is Its Own Beast — the memory math: weights + gradients + optimizer state + activations, and the ~16–20 bytes/param rule that puts a 7B model’s training state past 100 GB. Forward + backward, activation checkpointing, and gradient accumulation.
2. Parallelism: Data, Tensor, Pipeline, FSDP — when one GPU cannot hold the job, you split it across many. Replicas, splitting matrices, splitting layers, and sharding the optimizer (ZeRO/FSDP) — plus the communication tax each one pays.
3. LoRA, QLoRA & PEFT — the cheap path. Freeze the base, train a low-rank adapter that is under 1% of the parameters, and fine-tune a 65B model on a single GPU.
4. Fine-tune vs RAG vs Prompt — the decision most readers actually face. When to train at all, and when a prompt or a retrieval pipeline is the cheaper, faster answer.

The question to keep asking

On every page: what does this cost in latency, memory, and dollars, and how do we make it cheaper? In training the dominant term is GPU-hours × memory pressure — and the cheapest win is usually to train far fewer parameters, not to buy more GPUs.

Scope

We focus on fine-tuning and the efficiency techniques around it, not the full art of pretraining (data curation, learning-rate schedules, loss curves). If you operate AI systems, fine-tuning is the part of training you are most likely to touch.

By the end you will be able to look at a model size and a goal — “adapt this 13B to our support tickets” — and say what it will cost: how much GPU memory, full fine-tune or LoRA, one card or eight, and whether you should be training at all instead of writing a better prompt.

It actually happened — why “almost nobody pretrains from scratch”

The claim that opens this part has a concrete origin. When OpenAI trained GPT-3 (175 billion parameters) in 2020, the run reportedly spanned thousands of GPUs for weeks, with compute costs widely estimated in the millions of dollars — and that was before the even larger frontier models that followed. A pretraining run at that scale is a one-time, capital-heavy gamble that only a handful of organizations can finance, which is exactly why this part de-scopes it: the train-once economics put pretraining out of reach for almost every team. Fine-tuning inverts the bargain — start from someone else’s multi-million-dollar base model and spend a few GPU-hours steering it. The rest of this part is about making that spend as small as possible.

Check your understanding

1. Why does training get its own part instead of being folded into the inference chapters?
2. Roughly how much more memory does training need than inference, and where does the extra come from?
3. In the training regime, which resource is usually the binding constraint?
4. What do most of the techniques in this part have in common as a cost strategy?
5. Why does the part deliberately skip most of pretraining?

Show answers

1. Training is a different cost regime: it computes weight updates and must store gradients, optimizer state, and activations, so its costs (memory-bound, GPU-hours, no per-example latency) behave differently from inference.
2. Roughly 8–10× more — the extra comes from gradients, optimizer state (e.g. Adam moments), and the activations saved during the forward pass for the backward pass.
3. Memory — training state for even a mid-size model exceeds a single GPU, so fitting the job is the first problem.
4. They mostly avoid paying the full bill: shard the state so it fits across GPUs, or freeze most weights and train only a tiny fraction.
5. Because almost no reader will pretrain from scratch; the practical, money-saving skills are in fine-tuning and the efficiency techniques around it.