From chip to data centre

The ladder from one GPU package to a hall of racks: how many GPUs each rung holds, and how fast one GPU can move data at each.

Loading the animation…

Concept

A training or inference cluster is built in rungs, each a box of the rung below. The animation climbs them for GB200 NVL72 racks, arranged the way NVIDIA's published DGX SuperPOD reference architecture arranges them:

The package: one GPU, two compute dies and their HBM, reading memory at 8 TB/s.
The compute tray: 4 GPUs and two Grace CPUs on one 1U tray.
The rack: 18 compute trays and 9 NVLink switch trays. All 72 GPUs share one NVLink domain, so a GPU reaches any other in the rack at 900 GB/s per direction, exactly as fast as it reaches its neighbour on the same tray.
The scalable unit: 8 racks, 576 GPUs, joined by an InfiniBand fabric. Each GPU has one 400 Gb/s network port: 50 GB/s per direction.
The SuperPOD: up to 16 scalable units, 9,216 GPUs.
The hall: sized by power. At about 120 kW per rack, 100 MW of rack power is 833 racks, 59,976 GPUs.

Watch the bars as the camera zooms out. The rack boundary is the cliff: crossing it, a GPU's bandwidth falls 18 times, and beyond it, it never recovers. One gigabyte takes 125 µs to read from HBM, 1.11 ms to send over NVLink and 20 ms to send over the network.

That one ratio shapes the rest of the design. Traffic that is heavy and frequent (tensor-parallel all-reduces in every layer, expert-parallel all-to-alls, KV caches handed from prefill to decode) is kept inside the NVLink domain. Traffic that is lighter or can overlap with compute (data-parallel gradient averaging, pipeline-stage activations) crosses the network. The rest of this site follows that line: what is inside the package (chapter 2), on the tray (chapter 3), in the rack (chapter 4) and between racks (chapter 5).

Maths

At rung $r$ with $N_r$ GPUs and bandwidth $B_r$ per GPU per direction, moving $S$ bytes out of one GPU takes, as a bandwidth term only,

t_r = \frac{S}{B_r}, \qquad \text{the rung's aggregate is } N_r B_r, \qquad \text{the drop from the rung below is } \frac{B_{r-1}}{B_r}.

For GB200 the per-GPU figures are $B_{\text{HBM}} =$ 8 TB/s, $B_{\text{NVLink}} = \tfrac12 \times$ 1.8 TB/s, and $B_{\text{network}} =$ 400 Gb/s $\div 8$ . The rack's power per GPU is the rack's power divided by its 72 GPUs, so the power of a rung is $N_r \times P_{\text{rack}} / 72$ ; the hall holds $\lfloor 100\ \text{MW} / P_{\text{rack}} \rfloor$ racks.

Code

The rungs' derived figures, cut from src/lib/racks/model.ts (the Python reference, reference/rack_model.py, is the same, and the unit tests require identical results):

return rungs.map((r, i) => ({
  ...r,
  i,
  time: payload / r.bw,
  power_w: r.gpus * perGpuW,
  aggregate: r.gpus * r.bw,
  drop: i > 0 ? rungs[i - 1]!.bw / r.bw : 1.0,
}));

What the rack is made of

	GB200 NVL72
GPUs	72[gb200-page]
CPUs	36[gb200-page]
Compute trays	18[dgx-gb200-guide]
GPUs per tray	4[dgx-gb200-guide]
NVLink switch trays	9[dgx-gb200-guide]
Switch chips per tray	2[dgx-gb200-guide]
NVLink, whole rack	130 TB/s[gb200-page]
HBM, whole rack	13.4 TB[gb200-page]
HBM bandwidth, whole rack	576 TB/s[gb200-page]
Network per GPU	400 Gb/s[dgx-gb200-guide]
Rack power	120 kW[dgx-gb200-guide]
cooling	liquid to cold plates on CPUs and GPUs; networking and storage air-cooled[dgx-gb200-guide]

The same rack shape carries the next GPUs: GB300 NVL72 keeps 72 GPUs and 130 TB/s of NVLink, and Vera Rubin NVL72, whose NVLink figures NVIDIA marks preliminary, raises the rack's NVLink to 216 TB/s. NVIDIA's reference for a Vera Rubin AI factory puts 40,000 GPUs in 100 MW.