Parallelism: Data, Tensor, Pipeline, FSDP

The previous page showed a 7B model needing 112 GB of training state — past any single GPU. When one card cannot hold the job, you spread it across many. You have already met some of these splits in Tensor & Pipeline Parallelism for serving; training reuses the same geometry but adds a wrinkle — the optimizer state — and that wrinkle is what FSDP exists to solve. The throughline here is sharp: every split that saves memory buys it by sending data over a wire, and wires are slower than math.

The four strategies

Data parallelism (DP). Put a full copy of the model on every GPU, feed each a different slice of the batch, then all-reduce the gradients so every replica applies the same update. Simple and the throughput workhorse — but it saves no memory: each GPU still holds the entire 112 GB of state. DP scales speed, not size. Communication: one all-reduce of the full gradient each step.

Tensor parallelism (TP). Split individual weight matrices across GPUs — GPU 0 holds the left half of a matrix, GPU 1 the right — so each card stores and computes a fraction of every layer. This cuts memory per GPU, but it communicates inside every layer, on every forward and backward pass (an all-reduce per layer). That is a brutal amount of chatter, so TP only pays off over the fastest link you have — NVLink, inside a single node.

Pipeline parallelism (PP). Split the model by layers — GPU 0 holds layers 1–8, GPU 1 holds 9–16, and so on — and pass activations down the line. Communication is light (just the activations at each boundary), so PP tolerates the slower network between nodes. The catch is the pipeline bubble: while GPU 0 works on the first micro-batch, GPUs downstream sit idle until the work reaches them. Fine-grained micro-batching keeps the bubble small.

ZeRO / FSDP — shard the state itself. The insight: in plain data parallelism, every GPU stores an identical copy of the optimizer state, gradients, and parameters — pure redundancy. ZeRO (and PyTorch’s FSDP, Fully Sharded Data Parallel) shards those across the N GPUs so each holds only 1/N. When a layer is needed, an all-gather reconstructs its full parameters just-in-time; after the layer runs, the gathered copy is freed. Gradients are combined with a reduce-scatter so each GPU ends up with only its shard. You keep the simplicity of data parallelism but pay only 1/N of the memory.

Plain DP (per GPU):     [ full params | full grads | full optimizer ]  × N copies
FSDP / ZeRO-3 (per GPU):[ 1/N params  | 1/N grads  | 1/N optimizer  ]  + transient gather

Worked example: 7B on 8 GPUs

The 112 GB of training state, two ways across 8 GPUs:

Data parallel:   each GPU still holds   112 GB   → needs 8 × 80 GB+ cards, wasteful
FSDP (ZeRO-3):   each GPU holds  112 / 8 = 14 GB  + one gathered layer + activations
                 → fits on 8 × 24–40 GB cards

Same eight GPUs, same job — but sharding turns “needs eight 80 GB monsters” into “fits on eight commodity cards,” because nothing is stored eight times over. That is the entire reason FSDP is the modern default for fine-tuning models that don’t fit on one GPU.

3D parallelism

The giant pretraining runs combine all three axes — data × tensor × pipeline (often with ZeRO on top). Tensor-parallel within a node (fast NVLink absorbs the per-layer chatter), pipeline-parallel across nodes (only activations cross the slow network), and data-parallel over the whole cluster to scale throughput. Choosing the split is a placement puzzle: put the chattiest communication on the fastest wire.

Strategy	Splits	Saves memory?	Communication	Lives on
Data (DP)	nothing (full replica)	No	All-reduce gradients, once/step	Anywhere
Tensor (TP)	weight matrices	Yes	All-reduce per layer (heavy)	Inside a node (NVLink)
Pipeline (PP)	layers	Yes	Activations at boundaries (light)	Across nodes
FSDP / ZeRO	params + grads + optimizer	Yes (1/N)	All-gather + reduce-scatter	Anywhere; default for fine-tuning

The trade in one sentence

Memory saved per GPU goes up and communication volume goes up together — you are always paying for smaller-per-card with more-traffic-over-the-wire.

Reach for FSDP first

For most fine-tuning, FSDP (or DeepSpeed ZeRO) on a handful of GPUs is the right starting point: it shards the expensive optimizer state, needs no model-surgery, and runs over ordinary interconnect. Tensor and pipeline parallelism are tools you add when a single layer or the activation memory is itself too big to shard simply.

It actually happened — Megatron-LM and ZeRO (2019–2020)

The strategies on this page have named origins in the scramble to train models too big for any GPU. Megatron-LM (NVIDIA, 2019) introduced the tensor-parallel split — slicing individual weight matrices across GPUs — that this page describes, and showed it working on multi-billion-parameter transformers over NVLink. ZeRO (the Zero Redundancy Optimizer, Microsoft, 2020), shipped in the DeepSpeed library, contributed the sharding insight directly: it eliminated the redundant per-GPU copies of optimizer state, gradients, and parameters in stages (ZeRO-1, -2, -3), making it possible to train models like the 17B Turing-NLG and, later, far larger ones. PyTorch’s FSDP is essentially ZeRO-3’s parameter-and-optimizer sharding folded into the framework — which is why it’s the modern fine-tuning default this page recommends. The giant runs combine all of it, exactly the “put the chattiest communication on the fastest wire” placement puzzle described above.

The architect’s lens

Four strategies, one law — every split trades per-card memory for traffic over a wire:

• Why does it exist? A 7B model’s 112 GB of training state outgrows any single GPU, so when one card can’t hold the job you spread it — and each strategy splits a different thing (the batch, weight matrices, layers, or the optimizer state).
• What problem does it solve? Fitting (and speeding) a job no card can hold: FSDP/ZeRO shards params + grads + optimizer to 1/N (14 GB/GPU for 7B on 8 cards), turning “needs eight 80 GB monsters” into “fits on eight commodity cards.”
• What are the trade-offs? Memory-saved and communication-volume rise together — tensor parallelism all-reduces inside every layer (NVLink-only, within a node), pipeline parallelism is light but adds a bubble, FSDP adds an all-gather + reduce-scatter.
• When should I avoid it? Plain data parallelism saves no memory (a full replica per GPU) — use it only to scale throughput; reach past FSDP to TP/PP only when a single layer or the activation memory is itself too big to shard simply.
• What breaks if I remove it? Without any split a too-big model simply won’t fit on the GPUs you can rent; drop FSDP back to plain DP and you waste up to 8× the memory storing identical copies.

Check your understanding

1. Why does plain data parallelism save no memory, and what does it scale instead?
2. Why must tensor parallelism stay within a single node, while pipeline parallelism can span nodes?
3. What redundancy does ZeRO/FSDP eliminate, and what communication does it add to do so?
4. For a 7B model on 8 GPUs, roughly how much state sits on each GPU under FSDP, and why does that beat data parallelism?
5. What is the pipeline bubble, and how do you shrink it?

Show answers

1. Each GPU holds a full copy of the model, gradients, and optimizer state, so per-GPU memory is unchanged; it scales throughput (examples/sec) by processing different batch slices in parallel.
2. Tensor parallelism all-reduces inside every layer on every pass — enormous traffic — so it needs the fastest link (NVLink within a node). Pipeline parallelism only passes activations at layer boundaries, light enough to tolerate the slower between-node network.
3. It eliminates the redundant identical copies of params, gradients, and optimizer state that plain DP keeps on every GPU; it adds an all-gather (to reconstruct each layer’s params just-in-time) and a reduce-scatter (to combine gradients into per-GPU shards).
4. About 112 / 8 = 14 GB per GPU plus a transiently gathered layer and activations — so it fits on commodity 24–40 GB cards, whereas data parallelism would require each GPU to hold the full 112 GB.
5. The idle time while early pipeline stages wait for work to reach later stages (and vice versa); splitting the batch into many small micro-batches keeps all stages busier and shrinks the bubble.