Multi-GPU & Multi-Node Serving

Tensor & pipeline parallelism gave you three ways to split a model and one rule: the frequency of communication decides which wire it can ride. This page combines them for a model that fits on neither one GPU nor one node — the frontier-scale case. The governing idea, straight from the interconnect page, is that the network between nodes is an order of magnitude slower than the NVLink inside them, so the entire layout exists to keep the chatty traffic on the fast wire and let only the quiet traffic cross the slow one.

The combined layout: TP inside, PP across

The standard frontier-serving topology is tensor parallelism within each node, pipeline parallelism between nodes. TP’s frequent all-reduce stays on NVLink; PP’s rare activation handoff is the only thing that crosses the network.

        NODE A (NVLink)                 NODE B (NVLink)
   ┌───────────────────────┐      ┌───────────────────────┐
   │ G0 G1 G2 G3 G4 G5 G6 G7│  PP  │ G0 G1 G2 G3 G4 G5 G6 G7│
   │ layers 1-40, TP=8      │ ───► │ layers 41-80, TP=8     │
   │ all-reduce on NVLink   │ net  │ all-reduce on NVLink   │
   └───────────────────────┘      └───────────────────────┘
        ▲ fast, chatty                ▲ fast, chatty
        └────── slow, quiet network hop between stages ──────┘

Total GPUs = TP × PP. Here 8 × 2 = 16 GPUs across 2 nodes. The single inter-node hop per pipeline boundary moves only an activation tensor — kilobytes to a megabyte — which the network tolerates. Every layer’s all-reduce, the expensive part, never leaves NVLink.

The KV cache splits too

It is not only the weights that shard. Under tensor parallelism the KV cache is partitioned across the TP GPUs along the head dimension — each GPU stores the keys/values for the attention heads it owns. Under pipeline parallelism, each stage holds KV only for its layers. So the KV budget you computed in Part 3 is spread across the whole group, which is what lets a model far larger than one GPU still keep a usable batch resident. The scheduler’s free-KV accounting now sums across ranks.

Placement is a first-class decision

Because crossing the network costs ~16× a NVLink hop (previous page’s worked number), where each rank physically sits matters as much as the math. Two GPUs that all-reduce together must be in the same NVLink domain; pipeline neighbors should be on nodes with the fewest network switches between them. Get placement wrong — say, a TP group accidentally split across two nodes — and the all-reduce that should take 0.4 ms takes 6.4 ms, and every token slows accordingly. Schedulers and orchestrators expose topology hints precisely so the fast-communicating ranks land on the fast wire.

The network is the bottleneck the moment you span nodes

Inside a node, NVLink keeps cross-GPU traffic nearly free. The instant the model spans nodes, the slowest hop any token must take caps throughput, and the all-reduce/handoff pattern must be designed around it. Minimizing cross-node traffic is not an optimization here — it is the difference between a model that serves and one that crawls.

Collective operations, briefly

The communication primitives have names worth knowing. All-reduce (sum partial results so every GPU ends with the full value) is TP’s workhorse, run after each layer. All-gather and reduce-scatter are its halves, used in some sharded schemes. Send/recv (point-to-point) is PP’s quiet activation handoff. The cost model is simple: all-reduce is O(data) per call but happens constantly, so it demands NVLink; send/recv happens rarely, so it tolerates the network. Everything on this page follows from that asymmetry.

Worked example: serving a 405B model on H100-class GPUs

How do you place a 405B-parameter model on 80 GB H100-class GPUs? Start from the capacity wall.

weights (FP16)      = 405e9 × 2 bytes = 810 GB
usable / GPU        ≈ 80 GB − ~10 GB (activations, KV, runtime) ≈ 70 GB
GPUs for weights    = 810 / 70 ≈ 11.6  → need 12+ GPUs

One node = 8 GPUs (NVLink).  12 > 8  → the model MUST span ≥ 2 nodes.

Layout: TP=8 inside each node, PP=2 across two nodes  →  8 × 2 = 16 GPUs
  • 16 × 70 GB ≈ 1120 GB usable  ≫  810 GB weights → room for KV + batch
  • each node holds half the layers (TP=8 on NVLink)
  • exactly ONE inter-node activation hop per token (the PP boundary)

Now the dollar lens. At ~$3/GPU-hour, 16 H100s ≈ $48/hour. If the cluster sustains, say, 4,000 tokens/sec across the batch:

$48/hr ÷ 3600 s = $0.01333 /s
per token = $0.01333 ÷ 4000 ≈ $0.0000033  → ~$3.33 per 1M tokens

Two design choices move that number hard. Quantize to INT8 (Part 4): 405 GB of weights → ~6–7 GPUs → fits in a single node, deleting the inter-node hop entirely and roughly halving the GPU bill. Bad placement (a TP group split across nodes) does the opposite: the per-token network tax multiplies and throughput — the denominator — collapses, so dollar-per-token climbs even though the GPU count is unchanged. Same 16 GPUs, very different invoice, decided entirely by layout.

It actually happened — Llama 3.1 405B (2024)

The “405B model” in the worked example is not hypothetical. Meta released Llama 3.1 405B in 2024, one of the first frontier-scale models shipped with open weights anyone could download and serve. At ~405B parameters in FP16 (~810 GB) it lands exactly in the regime this page describes: it does not fit in one 8-GPU node, so a from-scratch FP16 deployment must span nodes and pay the inter-node hop. Meta also shipped an official FP8 recipe — precisely because halving the weights pulls the model back inside a single node of 8 H100-class GPUs, the same “quantize to delete the network hop” move the worked example makes. The open release turned multi-node serving from a hyperscaler-only concern into something ordinary teams had to reason about.

Under the hood — NCCL is the library actually moving the bytes

The collective names — all-reduce, all-gather, reduce-scatter — are abstractions; on NVIDIA hardware the thing that runs them is NCCL (NVIDIA Collective Communications Library). NCCL is topology-aware: at startup it probes which GPUs share NVLink, which share a PCIe switch, and which are reachable only over the network, then picks an algorithm to match — typically a ring for large messages (bandwidth-optimal; a ring all-reduce moves ≈2× the data over 2(N−1) steps) and a tree for small ones (latency-optimal). This is the machinery behind the page’s topology hints: hand NCCL a layout where the ring stays on NVLink and the collective flies; force the ring across a node boundary and every step inherits the slow wire. The library is smart, but it can only route over the fabric you placed the ranks on.

The architect’s lens

Multi-node serving is what you reach for when a model fits on neither one GPU nor one node. The five questions:

• Why does it exist? Because a frontier model exceeds even a whole node — 405B in FP16 is 810 GB, needing 12+ GPUs while one NVLink node holds 8 — so you combine tensor parallelism inside each node with pipeline parallelism across nodes.
• What problem does it solve? Serving a model too big for one node while keeping TP’s frequent, synchronous all-reduce on fast NVLink and letting only PP’s rare activation handoff (kilobytes to a megabyte) cross the slow inter-node network.
• What are the trade-offs? Total GPUs = TP × PP (16 here, ~$48/hr), you pay one inter-node hop per token, and placement is first-class — a TP group accidentally split across nodes turns a 0.4 ms all-reduce into ~6.4 ms (~16×) and throughput collapses.
• When should I avoid it? Whenever quantization can pull the model back inside one node — INT8 drops 405B to ~6–7 GPUs, deleting the inter-node hop and roughly halving the bill — span nodes only when you truly must.
• What breaks if I remove it? You are capped at about one node’s worth of weights; any model larger than ~8 GPUs of HBM is simply unservable, so frontier-scale models have no deployment path.

Check your understanding

1. What is the standard frontier topology, and which parallelism rides which wire?
2. If TP=8 and PP=4, how many GPUs and how many 8-GPU nodes does the deployment use?
3. How does the KV cache split under tensor and pipeline parallelism?
4. Why is physical placement of ranks a first-class concern, not an afterthought?
5. In the worked example, why must the 405B FP16 model span two nodes, and how does INT8 quantization change the picture?

Show answers

1. Tensor parallelism within each node (its frequent all-reduce stays on fast NVLink) and pipeline parallelism across nodes (its rare activation handoff is the only traffic crossing the slow network).
2. TP × PP = 8 × 4 = 32 GPUs, across 4 nodes of 8 GPUs each (one node per pipeline stage).
3. Under TP the KV cache is partitioned across GPUs by attention head (each GPU stores KV for its heads); under PP each stage stores KV only for the layers it holds — so the total KV budget is spread across all ranks.
4. Because an inter-node hop costs ~16× a NVLink hop; ranks that all-reduce together must share a NVLink domain, and misplacing a TP group across nodes turns a 0.4 ms all-reduce into ~6.4 ms, slowing every token.
5. 405B FP16 = 810 GB needs ~12 GPUs at ~70 GB usable each, but one NVLink node holds only 8, so it must span ≥2 nodes and pay the inter-node hop; INT8 halves weights to ~405 GB → ~6–7 GPUs, which fits in a single 8-GPU node, removing the cross-node hop and roughly halving the GPU bill.