Tensor & Pipeline Parallelism

Interconnect & multi-GPU established the wall: when a model’s weights exceed one GPU’s HBM you must split it across many, and the wire between GPUs becomes the bottleneck. That page asked whether to go multi-GPU. This page asks how — there are three fundamentally different ways to split, each paying a different communication cost on a different wire. Pick wrong and you rent N GPUs to get 2× the speed; pick right and the cost-per-token barely rises.

Three axes of splitting

Tensor parallel (TP):  split each weight MATRIX across GPUs
   layer L  →  [GPU0 half] [GPU1 half]   all-reduce after every layer

Pipeline parallel (PP): split LAYERS across GPUs
   GPU0: layers 1-20  →  GPU1: layers 21-40  →  GPU2: layers 41-60
   activations flow forward, GPU-to-GPU, once per boundary

Data parallel (DP):    REPLICATE the whole model
   GPU0: full model  |  GPU1: full model   different requests, no model split

The one-line distinction: TP and PP make a too-big or too-slow model fit and compute; DP makes a model that already fits serve more traffic. They are composable — real deployments use TP and PP together (next page) and DP for replicas.

Tensor parallelism: split the matrix

Every transformer layer is a few big matrix multiplies. Tensor parallelism slices each weight matrix across TP GPUs: each GPU holds a fraction of the weights and computes a fraction of the output. But the layer’s output is only correct once the fractions are combined — so after (essentially) every layer the GPUs run an all-reduce, summing partial results across all TP ranks.

Y = X · W,  W split into row blocks W0, W1 across GPU0, GPU1 (X sliced to match)
  GPU0 computes X0·W0 (partial)   GPU1 computes X1·W1 (partial)
  all-reduce ──► every GPU now holds the full, summed Y   (then next layer)

That all-reduce happens dozens of times per token (once or twice per layer × many layers). It is small per call but constant and synchronous — fast cores stall waiting for the slowest GPU’s data to arrive. So TP is only viable on a fast, low-latency fabric: NVLink, inside one node. Run TP across the slow inter-node network and the all-reduce tax swamps the compute. Rule of thumb: TP stays intra-node, bounded by the GPUs on one NVLink server (typically 8).

Pipeline parallelism: split the layers

Pipeline parallelism puts different layers on different GPUs — GPU0 runs layers 1–20, hands the activations to GPU1 for 21–40, and so on. The communication is tiny: just the activation tensor crossing one GPU boundary per stage, point-to-point, no all-reduce. That small, infrequent transfer is why PP scales across nodes where TP cannot.

The cost is the pipeline bubble. With one request in flight, while GPU0 works, GPUs 1–3 sit idle waiting for its output; the pipeline fills and drains, and that idle time is pure waste.

Naive pipeline (1 batch) — shaded = idle:
  GPU0 |###|...|...|...|     fills first
  GPU1 |...|###|...|...|
  GPU2 |...|...|###|...|
  GPU3 |...|...|...|###|     drains last   ← bubble = (stages-1) of idle

Micro-batched — split the batch into 4 chunks to keep stages busy:
  GPU0 |#1|#2|#3|#4|..|..|
  GPU1 |..|#1|#2|#3|#4|..|
  GPU2 |..|..|#1|#2|#3|#4|
  GPU3 |..|..|..|#1|#2|#3|   bubble shrinks as micro-batches ↑

The fix is micro-batches: split the batch into many small chunks and stream them through so every stage almost always has work. The bubble fraction falls roughly as (stages − 1) / (stages − 1 + micro-batches) — more micro-batches, smaller bubble, but more bookkeeping.

Data parallelism: replicate for throughput

Data parallelism does not split the model at all — it runs full copies on separate GPUs and routes different requests to each. It buys throughput, not capacity: it cannot help a model that does not fit on one GPU. In serving, DP is just “more replicas,” which is the autoscaling story two pages on.

When to use which

Strategy	Splits	Communication	Wire	Use when
Tensor (TP)	each weight matrix	all-reduce per layer (frequent)	NVLink, intra-node	model too big for 1 GPU; need low latency
Pipeline (PP)	layers into stages	activations per boundary (rare)	network, cross-node	model too big for 1 node
Data (DP)	nothing (replicas)	none (independent)	any	model fits; need more throughput

The communication tax decides the wire

TP’s all-reduce is frequent and synchronous → it must ride NVLink → stays inside a node. PP’s handoff is rare and small → it tolerates the network → spans nodes. This is not a preference; it is forced by how often each strategy makes GPUs talk. Match the chattiness of the split to the speed of the wire and you keep the scaling-inefficiency tax — the gap between “N GPUs” and “N× performance” — small.

Worked example: the all-reduce cost of TP

Run TP=8 on one node. Each all-reduce moves the hidden-state activation: batch 32, hidden 8192, FP16 = 2 bytes.

activation size  = 32 × 8192 × 2 bytes ≈ 0.5 MB
all-reduce moves ≈ 2 × 0.5 MB ≈ 1 MB per call  (ring all-reduce ≈ 2× the data)
calls per token  ≈ 2 × 80 layers = 160
traffic / token  ≈ 160 × 1 MB ≈ 160 MB

On NVLink ~400 GB/s:  160 MB ÷ 400 GB/s ≈ 0.4 ms / token  (tolerable)
On network ~25 GB/s:  160 MB ÷ 25  GB/s ≈ 6.4 ms / token  (16× worse — TP across nodes is a disaster)

Same operation, same data — but the wire it crosses changes the per-token tax by ~16×. That single ratio is why TP lives on NVLink and PP crosses the network.

It actually happened — Megatron-LM (2019)

The tensor-parallel scheme on this page is Megatron-LM (Shoeybi et al., NVIDIA, 2019) almost verbatim. Megatron showed how to split a transformer’s two big sub-blocks — the attention projections and the MLP — across GPUs so that each block needs only a single all-reduce to stitch its output back together, and trained multi-billion-parameter models years before that was routine. It also established the rule this page lives by: because that all-reduce fires constantly and synchronously, Megatron-style TP is meant to run inside one NVLink server. Almost every modern engine’s tensor_parallel_size knob is configuring Megatron’s partition — the name faded into a parameter, but the design is the one you’re setting.

Under the hood — where the “×2” in the worked example comes from

The worked example counts 2 × 80 layers = 160 all-reduces per token — why exactly two per layer? Megatron’s partition is arranged so each transformer block synchronizes only once. In the MLP, the first weight matrix is split column-wise (each GPU computes a full slice of the hidden dimension with no communication) and the second row-wise (whose partial sums need one all-reduce). The attention block is handled the same way: heads are divided across GPUs, computed independently, and the output projection takes one all-reduce. Two big sub-blocks, one sync each → two all-reduces per layer in the forward pass. That column-then-row ordering is exactly what holds TP’s communication to the bare minimum the fast wire can absorb; a naïve split would sync after every matmul and double the tax.

The architect’s lens

Three ways to split a model, three communication taxes — picking right is the difference between N× and 2×. The five questions:

• Why does it exist? Because a model can be too big or too slow for one GPU, and there are three fundamentally different splits — TP (slice each weight matrix), PP (split layers into stages), DP (replicate the whole model) — each paying a different communication cost on a different wire.
• What problem does it solve? TP and PP make a model that won’t fit fit and compute; DP makes a model that already fits serve more traffic. They compose: TP+PP for capacity, DP for replicas.
• What are the trade-offs? TP runs an all-reduce dozens of times per token, synchronously — so it must ride NVLink and stay intra-node; PP’s handoff is rare so it spans nodes but pays the pipeline bubble (shrunk with micro-batches). The same all-reduce is ~16× costlier across the network than on NVLink (6.4 ms vs 0.4 ms/token).
• When should I avoid it? Use DP, not TP/PP, when the model already fits on one GPU — splitting adds a communication tax and latency for capacity you don’t need; never run TP across nodes, and never run PP with too few micro-batches.
• What breaks if I remove it? You are capped at models that fit on a single GPU; and if you split but mismatch the strategy to the wire (TP over the network), you rent N GPUs to get barely 2× the speed — the scaling-inefficiency tax.

Check your understanding

1. State the one-line difference between what TP/PP solve and what DP solves.
2. Why must tensor parallelism stay inside a single NVLink node?
3. What is the pipeline bubble, and how do micro-batches shrink it?
4. Pipeline parallelism communicates far less than tensor parallelism — so why not always use PP?
5. In the worked example, why is the same all-reduce ~16× more expensive across nodes than on NVLink, and what design rule does that justify?

Show answers

1. TP and PP make a model that is too big or too slow for one GPU fit and compute; DP replicates a model that already fits to serve more traffic (throughput, not capacity).
2. TP runs an all-reduce after essentially every layer — dozens of synchronous communications per token — so it needs a fast, low-latency fabric; on the slow inter-node network the all-reduce tax would swamp the compute.
3. The bubble is the idle time while the pipeline fills and drains, when downstream stages wait for upstream output; splitting the batch into many micro-batches keeps every stage busy, shrinking the bubble roughly as (stages−1)/(stages−1+micro-batches).
4. PP introduces the pipeline bubble and adds latency (a token traverses all stages in sequence), and it can only split along the layer dimension; TP gives lower per-token latency within a node and is needed when even one node’s worth of layers must be split further.
5. The all-reduce moves the same ~160 MB/token, but NVLink (~400 GB/s) clears it in ~0.4 ms while the network (~25 GB/s) takes ~6.4 ms — ~16× slower — justifying the rule that frequent-communication TP stays on NVLink intra-node and rare-communication PP crosses nodes.