Revision · Training & Fine-Tuning Efficiency

Part 7 crossed from the serve-forever world into the train-once regime. Its throughline: training is memory-bound and capital-heavy, so almost every technique here is about not paying the full bill — shard the state so it fits, or freeze most of the weights and train a sliver.

What this part covered

• Training is a different cost regime — it computes weight updates, so it must hold gradients, optimizer state, and saved activations on top of the weights; that pushes memory ~8–10× over inference, and latency-per-example simply doesn’t matter.
• The ~16 bytes/param rule — FP16 weights + FP16 gradients + FP32 master copy + Adam’s momentum and variance sum to ~16 bytes/param, which puts a 7B model’s persistent training state past 112 GB before a single activation — why the optimizer, not the weights, blows the budget.
• Memory-for-compute trades — activation checkpointing recomputes activations in the backward pass (O(layers) → O(√layers) memory for ~20–30% more compute), and gradient accumulation gets big-batch statistics by summing micro-batch gradients, paying only in wall-clock time.
• Parallelism spreads a job no card can hold — data (full replica, saves no memory, scales speed), tensor (splits matrices, heavy per-layer chatter, NVLink-only), and pipeline (splits layers, light traffic, tolerates the bubble) each split a different thing.
• FSDP / ZeRO is the fine-tuning default — sharding params + grads + optimizer to 1/N turns “needs eight 80 GB monsters” into “fits on eight commodity cards” (14 GB/GPU for 7B on 8), at the cost of an all-gather + reduce-scatter — the one law being that per-card memory saved and wire traffic rise together.
• LoRA / PEFT freezes the base — training a low-rank B·A adapter (~0.39% of a matrix at r=8, under 0.1% of a 7B model) collapses Adam state from ~84 GB to ~64 MB, because the update has low intrinsic rank.
• QLoRA removes the last fixed cost — quantizing the frozen base to 4-bit (NF4) fits a 65B fine-tune on a single 48 GB GPU; adapters stay swappable (merge for zero overhead, or hot-swap many tasks on one resident base).
• Fine-tune vs RAG vs prompt — form / facts / first: fine-tune changes weights (style, format, shortening prompts), RAG changes retrieved knowledge (fresh, citable facts), prompt changes the input (try first). They compose; don’t fine-tune to add facts — it memorizes poorly and can raise hallucination.

The takeaway

Training inverts the book’s usual bargain: memory is the wall you hit first, dollars come from GPU-hours, and the cheapest win is almost always to train fewer parameters rather than buy more GPUs. Size the training state, reach for FSDP or LoRA/QLoRA before a cluster, and — most often — ask whether a prompt or a retrieval pipeline would have been cheaper than any training at all. Part 8 turns to measuring whether these levers actually paid off.