Trays and nodes

A compute tray's GPUs, Grace CPUs and network cards, and the paths data takes between them: network, CPU memory, another GPU.

Loading the animation…

Concept

A GB200 NVL72 compute tray is a 1U box with two GB200 superchips: 2 Grace CPUs and 4 Blackwell GPUs. Each Grace connects to its two GPUs over NVLink-C2C (900 GB/s, both directions together) and has up to 480 GB of its own LPDDR5X memory at up to 512 GB/s. The tray also carries 4 ConnectX-7 network cards, one 400 Gb/s port per GPU for the compute fabric, and 2 BlueField-3 DPUs for storage and management traffic. The GPUs' NVLinks leave from the back of the tray through cable cartridges to the switch trays.

Data reaches a GPU's HBM by three kinds of path, and the animation runs each one with a 2 GB payload:

From the network (GPUDirect RDMA): the network card writes straight into GPU memory, without a copy through CPU memory. The port is the slow link: the whole path runs at 50 GB/s at best.
From Grace's memory over NVLink-C2C: the way a GPU uses CPU memory as an overflow for its KV cache or for offloaded weights. Here C2C sets the pace, 450 GB/s per direction: 4.7 ms for 2 GB in 16 chunks, against 42 ms from the network.
From another GPU in the rack over NVLink, through the switch trays: 900 GB/s per direction.

Switch the chunk count. Sent as one piece, a payload crosses each link in turn, so the times add up (8.6 ms from Grace). Cut into chunks, the links work at once on different chunks, and the transfer approaches the slowest link's time (94% of it with 16 chunks). Real transfer engines pipeline the same way, in much smaller pieces.

Maths

Chunk $c$ starts on link $i$ when it has left link $i-1$ and chunk $c-1$ has left link $i$ :

s_{c,i} = \max\left(f_{c,i-1},\ f_{c-1,i}\right), \qquad f_{c,i} = s_{c,i} + \frac{S/C}{B_i} .

For identical chunks this flow shop finishes at $T = \sum_i \frac{S/C}{B_i} + (C-1)\frac{S/C}{B_{\min}}$ : the first chunk fills the pipe, then the slowest link delivers one chunk per $\frac{S/C}{B_{\min}}$ . The tests check the recurrence against this closed form for every path and chunk count.

Code

The recurrence, cut from src/lib/racks/model.ts:

for (let c = 0; c < chunks; c++) {
  for (let i = 0; i < h; i++) {
    const a = i > 0 ? finish[c]![i - 1]! : 0.0;
    const b = c > 0 ? finish[c - 1]![i]! : 0.0;
    const s0 = a > b ? a : b;
    start[c]![i] = s0;
    finish[c]![i] = s0 + size / hops[i]!.bw;
  }
}

The 8-GPU node

Before rack-scale NVLink, the unit was the 8-GPU server: an HGX baseboard with its own NVLink switches, x86 host CPUs over PCIe, and one network card per GPU. It is still the common shape of an air-cooled cluster.

	DGX B200 (8-GPU node)
GPUs	8[dgx-b200-page]
HBM, whole rack	1.44 TB[dgx-b200-page]
HBM bandwidth, whole rack	64 TB/s[dgx-b200-page]
NVLink, whole rack	14.4 TB/s[dgx-b200-page]
NVSwitch chips	2[dgx-b200-page]
Network per GPU	400 Gb/s[dgx-b200-page]
System power	14.3 kW[dgx-b200-page]
Rack units	10[dgx-b200-page]

Its NVLink domain ends at the chassis: 8 GPUs share 14.4 TB/s, and everything beyond them goes over the network. The NVL72 rack moves that boundary out to 72 GPUs (chapter 4).