Interconnect & Why Big Models Need Many GPUs

Numeric Precision showed that a 70B model in FP16 needs ~140 GB — more than one 80 GB GPU holds. Quantizing helps, but the frontier models keep growing faster than HBM does, so eventually you have no choice: one model must live across many GPUs. The moment you do that, a new and slower resource enters the cost equation — the interconnect, the wiring that lets GPUs talk to each other. This page is about why that wiring, not the HBM, becomes the wall.

The wall: HBM capacity

Recall the capacity math. A parameter in FP16 is 2 bytes, so model weights need params × 2 bytes, and the GPU has a fixed ~80 GB:

70B  FP16 → 140 GB  >  80 GB   → won't fit on one GPU

When the weights exceed one card’s HBM, you shard them: put some layers (or some columns of each weight matrix) on GPU 0, the rest on GPU 1, and so on. Now a single forward pass has to hand intermediate activations from one GPU to another — and that handoff travels over a wire that is much slower than HBM.

The bandwidth cliff

Here is the hierarchy that matters once you go multi-GPU. Each step outward is dramatically slower than the last:

   HBM (on-package, GPU ↔ its own memory)     ~2-3.35 TB/s   ← from earlier pages
        │
        ▼
   NVLink (GPU ↔ GPU, same server/node)        ~hundreds of GB/s
        │
        ▼
   PCIe (GPU ↔ GPU without NVLink, or ↔ CPU)   ~tens of GB/s  (PCIe 4 x16 ≈ 32 GB/s)
        │
        ▼
   Network (node ↔ node, e.g. InfiniBand)      ~tens of GB/s per link, + real latency

• NVLink is NVIDIA’s dedicated GPU-to-GPU fabric inside a server. Fast — hundreds of GB/s — but still an order of magnitude below HBM.
• PCIe is the general-purpose bus to the CPU and, absent NVLink, between GPUs. ~32 GB/s for PCIe 4.0 x16 — roughly 100× slower than HBM.
• Network (InfiniBand/Ethernet) connects separate servers. This is inter-node; it adds latency on top of limited bandwidth.

The key distinction is intra-node vs inter-node. GPUs in the same chassis talk over NVLink (fast); GPUs in different chassis talk over the network (slow, with latency). This is why an 8-GPU server connected by NVLink is a sweet spot: you can split a model across 8 GPUs and still keep cross-GPU traffic on the fast fabric.

The interconnect is the new bottleneck

The instant a model spans multiple GPUs, the slowest wire any activation must cross caps your speed. Splitting a model also means GPUs must synchronize — fast cores wait on slow wires. The art of multi-GPU serving is partitioning the model so that as little data as possible crosses as slow a wire as possible. We unpack the strategies in tensor & pipeline parallelism.

Why this is a cost problem

Going multi-GPU is doubly expensive. First, you are renting N GPUs instead of one — dollars scale with N. Second, you don’t get N× the speed, because the GPUs now spend time shuffling activations over NVLink/PCIe/network and waiting on each other instead of computing. That gap between “N GPUs” and “N× performance” is scaling inefficiency, and minimizing it — keeping traffic on fast wires, overlapping communication with compute — is most of what a serving team does. The cheapest multi-GPU job is the one that needed the fewest GPUs (hence the relentless push toward lower precision to fit on fewer cards).

Worked example: GPUs for a 405B model in FP16

How many 80 GB GPUs does a 405B-parameter model in FP16 need just to hold the weights?

weight bytes   = 405 × 10^9 params × 2 bytes = 810 × 10^9 bytes = 810 GB

GPUs (weights) = 810 GB / 80 GB ≈ 10.1  → round up → 11 GPUs

But weights aren’t the whole story. You also need room for activations and the KV cache, plus CUDA/runtime overhead — comfortably another 10–20% or more. So in practice:

usable per GPU ≈ 80 GB − overhead ≈ ~70 GB
GPUs needed    = 810 GB / 70 GB ≈ 11.6 → 12+ GPUs

Twelve 80 GB GPUs do not fit in a single 8-GPU NVLink server, so this model spans two nodes — meaning some activations now cross the slow inter-node network every forward pass. That is exactly the regime where interconnect dominates latency.

Model / precision	Weight bytes	80 GB GPUs (weights only)	One 8-GPU node?
70B FP16	140 GB	2	Yes
70B INT8	70 GB	1	Single GPU
405B FP16	810 GB	~11 (12+ with overhead)	No — multi-node
405B INT8	405 GB	~6 (405/70 ≈ 5.8)	Yes (fits in 8)

The cost lens, one last time

Look at the 405B rows. In FP16 the model needs 12+ GPUs across two nodes, paying the inter-node network tax on every token. In INT8 it drops to ~6 GPUs and fits inside a single NVLink node — fewer GPUs rented and every cross-GPU hop stays on the fast fabric. The precision knob from the previous page directly buys you out of the most expensive tier of the interconnect hierarchy. Memory, latency, and dollars are the same problem wearing three hats.

Under the hood — NVSwitch and the real NVLink numbers

This page says NVLink runs at “hundreds of GB/s” — here are the actual figures, because the gap to PCIe is the whole reason an 8-GPU server is shaped the way it is. Per-GPU aggregate NVLink bandwidth is about 600 GB/s on the A100 (NVLink 3) and about 900 GB/s on the H100 (NVLink 4). Set that beside PCIe — roughly 32 GB/s for Gen4 x16, 64 GB/s for Gen5 — and NVLink is an order of magnitude faster for GPU-to-GPU traffic.

But raw link speed isn’t enough. With only point-to-point links, GPU 0 talking to GPU 7 might have to hop through intermediaries. The fix is NVSwitch: a switch chip (the same idea as a network switch) that wires every GPU in the node to every other at full NVLink bandwidth at once. That is what turns eight cards into a genuine all-to-all fabric, so tensor parallelism — which makes all the GPUs exchange activations on every layer — can run without one slow pair throttling the whole group.

This is why “fits in one 8-GPU NVLink node” is the magic phrase from the worked example. Inside the node, every cross-GPU hop rides NVSwitch at hundreds of GB/s; cross the node boundary and you drop to the inter-node network at tens of GB/s plus latency. The serving team’s job is to keep the chatty collective traffic (all-reduce and friends) on the fast side of that cliff.

The architect’s lens

The interconnect is the resource you adopt the instant a model outgrows one card — judge it with the five questions:

• Why does it exist? Because frontier weights exceed one GPU’s ~80 GB HBM — 70B FP16 is 140 GB, 405B is 810 GB — so the model must be sharded across GPUs, and every shard has to hand activations to the next over a wire.
• What problem does it solve? It lets a model too big for any single card fit and still compute one coherent forward pass, by spreading weights (and the KV cache) across many GPUs’ memory.
• What are the trade-offs? Every step outward is roughly an order of magnitude slower than HBM — NVLink ~hundreds of GB/s, PCIe ~32 GB/s, inter-node network tens of GB/s plus latency — so you rent N GPUs but get less than N× speed (scaling inefficiency), fast cores stalling on slow wires.
• When should I avoid it? Whenever you can stay on fewer GPUs: quantizing a 405B model to INT8 drops it from 12+ GPUs across two nodes to ~6 inside a single NVLink node — fewer GPUs rented and no inter-node tax.
• What breaks if I remove it? You are hard-capped at one card’s HBM — about 40B params in FP16 — and any larger model is simply unservable; there is no single-GPU path to a 405B model.

Check your understanding

1. Why must some models be split across multiple GPUs in the first place?
2. Rank HBM, NVLink, PCIe, and inter-node network by bandwidth, and state which boundary is “intra-node vs inter-node.”
3. Why does splitting a model across GPUs not give you a proportional N× speedup?
4. Compute the number of 80 GB GPUs needed to hold a 405B model’s weights in FP16, then explain why the real number is higher.
5. Using the final table, explain how INT8 lets the 405B model avoid crossing the inter-node network.

Show answers

1. Because the model’s weights exceed one GPU’s HBM capacity (e.g., 70B FP16 = 140 GB > 80 GB), so the weights must be sharded across several GPUs’ memory.
2. Fastest to slowest: HBM (~2–3.35 TB/s) > NVLink (~hundreds of GB/s) > PCIe (~tens of GB/s, ~32 GB/s for PCIe 4 x16) > inter-node network (~tens of GB/s plus latency). NVLink/PCIe within a server is intra-node; the network between servers is inter-node.
3. Because the GPUs must exchange activations over the slower interconnect and synchronize, so fast cores wait on slow wires — that scaling inefficiency means N GPUs deliver less than N× performance.
4. 405 × 10⁹ × 2 = 810 GB; 810 / 80 ≈ 10.1 → 11 GPUs for weights alone. The real number is higher because activations, KV cache, and runtime overhead consume usable HBM, so per-GPU usable space is ~70 GB, pushing it to ~12+.
5. INT8 halves bytes/param to 1, so 405B needs ~405 GB → ~6 GPUs (405/70 ≈ 5.8), which fits inside a single 8-GPU NVLink node; all cross-GPU traffic stays on fast NVLink instead of crossing the slow inter-node network.