Scale-up: NVLink and NVSwitch

72 GPUs, 18 switch chips, one link from every GPU to every chip: why an NVL72 rack behaves like one big GPU for all-reduce and all-to-all.

Loading the animation…

Concept

Many of the communications in a model split over GPUs are collectives: every GPU contributes and every GPU receives. Tensor parallelism ends each attention and MLP block with an all-reduce (every GPU needs the sum of everyone's partial results); expert parallelism routes tokens to the GPUs holding their experts with an all-to-all.

The animation runs the classic ring all-reduce on real numbers. Each GPU cuts its vector into $n$ chunks. In the reduce-scatter half, every GPU sends one chunk to its neighbour, which adds it to its own; after $n-1$ steps each GPU owns one complete chunk of the sum. In the all-gather half, the finished chunks travel round the ring for another $n-1$ steps. Every link is busy every step, and each GPU sends $2(n-1)/n$ of its vector, close to twice its size, whatever $n$ is: the ring is bandwidth-optimal (Patarasuk and Yuan, 2009).

So the time is set by the link. Over NVLink, at 900 GB/s per direction, a 64 MB all-reduce over 8 GPUs takes 124 µs; over the GPU's 400 Gb/s network port it takes 2.24 ms, 18 times longer. A tensor-parallel group does this twice per layer, every layer, so it is kept inside the NVLink domain.

NVLink switches can also do the reduction themselves: NVIDIA's switches since the third-generation NVSwitch add the vectors in the network (SHARP in-network reduction, with multicast), so each GPU sends its vector once and receives the sum once, 71.1 µs for the same 64 MB.

Small messages: the latency term

Loading the animation…

Concept

The ring above is timed by bandwidth alone, which is right for the gigabytes of a training step's gradients and wrong for the messages a decode step sends. Every step of a collective also pays a fixed cost: launching the transfer, synchronising with the neighbour, the trip through the switch. Call it $\alpha$ . The ring has $2(n-1)$ steps, so it pays $2(n-1)\alpha$ before a single byte's bandwidth counts; the in-switch reduction pays one fixed cost $\alpha_{\text{s}}$ .

The model takes $\alpha$ from measurement. On an eight-GPU NVLink 4 node, NCCL's all-reduce of small messages takes about 4.7 µs as a ring and 5.6 µs through the switches (NVLS), for anything up to 64 KiB (Ma et al., 2026, section VII-C). Read as $2(n-1)\alpha$ that is 336 ns per ring step. NVIDIA does not publish NVLink 5's latency, so the B200 curve uses the same $\alpha$ : an assumption, labelled in the widget.

Now a tensor-parallel decode step. Llama 3 70B's all-reduce at batch 8 is 131 kB (8 tokens of 8,192 16-bit values). On eight B200s it takes 4.95 µs as a ring, of which 95% is the fixed cost; the bandwidth-only formula says 255 ns, 19.4 times too little. Below the crossover, 2.42 MB for a ring of eight, doubling NVLink's bandwidth hardly helps; cutting the steps does. That is why the in-switch reduction and one-shot algorithms (every GPU reads everyone's data directly in one step) matter for decode, and why tensor parallelism stays inside the NVLink domain: the network adds its own latency as well as less bandwidth.

A measured check. For large messages the same paper measures NCCL's NVLS all-reduce at 276 GB/s on that node. The model's in-switch bound for a gigabyte there is 448.9 GB/s, so the bound is 1.63 times what NCCL reaches. The paper does not say whether its figure is NCCL's algorithm or bus bandwidth; if it were bus bandwidth, NCCL would reach less of the bound still. This is the site's first check of the model against a measurement rather than against the vendor's own arithmetic (the checks).

The NVL72 rack as one domain

Blue: GPU 0's 18 links, one to each switch chip; grey: the last GPU's. Every GPU is wired the same way, so every pair of GPUs is one switch hop apart (NVIDIA DGX GB200 user guide; GB200 NVL72 technical blog). Link speeds are bidirectional totals, as NVIDIA quotes them.

An NVL72 rack wires all 72 GPUs together through 18 switch chips. Each GPU has 18 NVLinks and each switch chip has 72 ports, so the GPUs' 1,296 links fill the chips' 1,296 ports exactly, one link from every GPU to every chip. Every pair of GPUs is one switch hop apart, at the full 1.8 TB/s per GPU (both directions), 129.6 TB/s across the rack (NVIDIA rounds it to 130 TB/s). This is what NVIDIA means by the rack acting as one GPU: for collectives, any 72-GPU group is as close as two GPUs on one tray.

It matters most for all-to-all. An expert-parallel layer across all 72 GPUs sends most of each GPU's tokens to other GPUs; 64 MB per GPU takes 70.1 µs inside the rack, and would take 1.26 ms over the network ports.

Maths

With $n$ GPUs, vector size $S$ and link bandwidth $B$ per direction, the ring sends $n-1$ chunks of $S/n$ in each half:

T_{\text{ring}} = 2\,\frac{n-1}{n}\,\frac{S}{B}, \qquad T_{\text{in-switch}} = \frac{S}{B}, \qquad T_{\text{all-to-all}} = \frac{n-1}{n}\,\frac{S}{B}.

The NVL72 wiring: $72 \times 18 = 1296$ GPU links and $18 \times 72 = 1296$ switch ports. A switch tray's two chips carry $2 \times 72 \times 50\ \text{GB/s} \times 8 =$ 57.6 Tb/s per direction, which is the figure in NVIDIA's DGX GB200 user guide.

Code

The reduce-scatter half, cut from src/lib/racks/model.ts:

for (let s = 0; s < n - 1; s++) {
  const sends: RingSend[] = [];
  for (let g = 0; g < n; g++) {
    const c = mod(g - s, n);
    sends.push({ from: g, to: (g + 1) % n, chunk: c, value: cur[g]![c]! });
  }

Three generations of NVLink domain

	H100 SXM5	Blackwell (GB200 NVL72)	Rubin (Vera Rubin NVL72)preliminary
NVLink per GPU (both directions)	900 GB/s[nvlink-page]	1.8 TB/s[nvlink-page]	3 TB/s[vr-page]
NVLink links per GPU	18[nvlink-page]	18[nvlink-page]	36[nvlink-page]

NVLink 4 (Hopper) connected 8 GPUs in one server through its own NVSwitches; NVLink 5 (Blackwell) reaches 72 GPUs in one rack; NVLink 6 (Vera Rubin, preliminary) keeps 72 GPUs and raises each GPU's links to 3 TB/s, 216 TB/s across the rack.