Scale-out: the network

Leaf, spine and core switches, rail-aligned fabrics, oversubscription, and the transceivers every link needs.

Loading the animation…

Concept

Past the rack, GPUs reach each other through a switched network: InfiniBand (NVIDIA Quantum) or Ethernet (Spectrum-X and others), one port per GPU. The standard shape is the fat tree, or folded Clos network (Leiserson 1985; Al-Fares, Loukissas and Vahdat 2008): leaf switches connect GPUs, spine switches connect leaves, and when that runs out, core switches connect groups of leaves and spines. If every leaf has as many links up as GPUs down, any half of the GPUs can talk to the other half at full rate: the fabric is non-blocking.

The switch's port count sets the size. Two tiers of $k$ -port switches reach $k \times k/2$ GPUs: 2,048 with 64-port Quantum-2 switches, 10,368 with 144-port Quantum-X800 switches (NVIDIA says its 144-port photonics switch connects "over 10,000 GPUs" in a non-blocking two-level fat tree). Beyond that a third tier is needed, and the longest path grows from 4 links to 6.

Build a fabric for 9,216 GPUs in the animation. With 64-port switches it takes three tiers and 720 switches; with 144-port switches, two tiers and 192. The transceivers follow the links: 55,296 against 36,864, six per GPU against four.

Oversubscription trades bandwidth for hardware: at 2:1 a leaf gives two-thirds of its ports to GPUs and one-third to uplinks, so 9,216 GPUs on 64-port switches need 556 switches instead of 720, but when every GPU sends across the fabric at once each gets half its port's rate.

Rail-aligned: the published SuperPOD design

NVIDIA's DGX SuperPOD reference architecture for GB200 does not build the generic tree. Its fabric is rail-aligned: a compute tray's four network ports go to four different leaf switches, one per rail, and each rack has its own leaf for each rail (8 racks × 4 rails = 32 leaves per scalable unit). Each rail group has 6 spines, and core switches join scalable units; in NVIDIA's words, traffic between different racks, or between different rails, traverses the spine layer. The model applies these rules and reproduces the switch counts of NVIDIA's published table exactly:

Check	Kind	Model	Published	Ratio
SuperPOD 1152 GPUs: leaf switches [superpod-network]	rule	64	64	1.0000
SuperPOD 1152 GPUs: spine switches [superpod-network]	rule	48	48	1.0000
SuperPOD 1152 GPUs: core switches [superpod-network]	rule	18	18	1.0000
SuperPOD 2304 GPUs: leaf switches [superpod-network]	rule	128	128	1.0000
SuperPOD 2304 GPUs: spine switches [superpod-network]	rule	96	96	1.0000
SuperPOD 2304 GPUs: core switches [superpod-network]	rule	36	36	1.0000
SuperPOD 4608 GPUs: leaf switches [superpod-network]	rule	256	256	1.0000
SuperPOD 4608 GPUs: spine switches [superpod-network]	rule	192	192	1.0000
SuperPOD 4608 GPUs: core switches [superpod-network]	rule	72	72	1.0000
SuperPOD 9216 GPUs: leaf switches [superpod-network]	rule	512	512	1.0000
SuperPOD 9216 GPUs: spine switches [superpod-network]	rule	384	384	1.0000
SuperPOD 9216 GPUs: core switches [superpod-network]	rule	144	144	1.0000

The rail-aligned fabric uses more switches than the generic tree of the same switches: 512 leaves, 384 spines and 144 cores for 9,216 GPUs, against 720 switches in all for the generic tree, because each leaf serves one rack's rail and uses 36 of its 64 ports. In exchange, each leaf's cabling stays with one rack, and a scalable unit's fabric is the same whether or not all its racks are installed: NVIDIA's design notes ask for the full unit's leaves and leaf-to-spine cables to be built even when some racks are missing.

Maths

With $k$ -port switches, leaves give $d$ ports down and $u = k - d$ up; non-blocking is $d = u = k/2$ , and $r{:}1$ oversubscription is $d = \lfloor k r / (r+1) \rfloor$ . Two tiers hold $N \le k\,d$ GPUs with

L = \left\lceil \frac{N}{d} \right\rceil, \qquad S = \left\lceil \frac{L\,u}{k} \right\rceil .

Three tiers group $k/2$ leaves and $k/2$ aggregation switches into pods of $k d / 2$ GPUs under $\lceil P\,(k/2)^2 / k \rceil$ core switches for $P$ pods; a full non-blocking tree holds $k^3/4$ GPUs with $5k^2/4$ switches (Al-Fares et al.). Links: $N$ to the leaves, $L u$ leaf-to-spine and, with three tiers, $P (k/2)^2$ aggregation-to-core; every optical link has a transceiver at each end.

Code

The two-tier case, cut from src/lib/racks/model.ts:

const down = Math.floor((k * oversub) / (oversub + 1));
const up = k - down;
const twoMax = k * down;

Optics: pluggable and co-packaged

Every switch-to-switch link here is optical, with a pluggable transceiver at each end, and at 9,216 GPUs a non-blocking fabric needs tens of thousands of them. Co-packaged optics move the optical engines into the switch package, removing the pluggable modules. NVIDIA quotes "3.5x power efficiency" for its co-packaged switches against pluggable transceivers; the claim is shown here, not used in the model.

Computing in the link. A further step is for the optical module to do arithmetic on the data it carries. The Compute in Transit results test one such case end to end: whether a photonic link module could compress training gradients as they cross the network. At the scale tested it could not do the compression that trains well (keeping the largest values needs comparisons), and the site asks why the same work should not be done by digital logic in the NIC.

Switch	Ports	Optics	Availability
Quantum-2 (QM9700 class), InfiniBand NDR	64[quantum2-page] × 400 Gb/s	pluggable	product
Quantum-X800, InfiniBand XDR	144[qx800-page] × 800 Gb/s	pluggable	product
Spectrum-4 SN5600, Ethernet	64[eth-switch-page] × 800 Gb/s	pluggable	product
Quantum-X Photonics Q3450, InfiniBand (co-packaged optics)	144[photonics-page] × 800 Gb/s	co-packaged	plannedcommercial availability planned for early 2026 (blog dated 2025-08-18)
Spectrum-X Photonics SN6810, Ethernet (co-packaged optics)	128[cpo-blog] × 800 Gb/s	co-packaged	plannedavailable in the second half of 2026
Spectrum-X Photonics SN6800, Ethernet (co-packaged optics, four ASICs)	512[cpo-blog] × 800 Gb/s	co-packaged	plannedavailable in the second half of 2026