Other designs

TPU pods built from torus cubes and optical switches, and other vendors' rack-scale systems: what each changes about the ladder.

Loading the animation…

Concept

NVIDIA's racks answer the scale-up question with switches: every GPU has a link to every NVLink switch chip, so any two GPUs in the rack are one switch hop apart (chapter 4). That is not the only answer.

Google's TPU pods: a torus and optical switches. A TPU chip links directly to its neighbours: six links, one each way in three dimensions. Sixty-four chips form a 4×4×4 cube wired with copper; the cubes are joined through optical circuit switches, which can be reconfigured to shape the pod into the torus a job needs. A TPU v5p pod has 8,960 chips, each with 1.2 TB/s of inter-chip bandwidth (both directions, Google's documentation); the TPU v4 paper reports that the optical switches and their optics are "<5% of system cost and <3% of system power". The TPU7x (Ironwood) pod has 9,216 chips.

A torus trades hops for wiring. In the animation, a message from one chip of a 4×4×4 cube reaches its six neighbours in one hop and the farthest chip in 6; the mean is 3.05 hops, against one in a switched rack. Neighbour traffic (a ring all-reduce, pipeline stages laid along a dimension) uses a torus well. All-to-all traffic, as in mixture-of-experts layers, is limited by the links a cut through the middle crosses: 32 for the cube. With each chip's bandwidth split equally over its six links (our assumption: Google states only the per-chip total), a 64 MB all-to-all within a cube takes at least 320 µs, against 70.1 µs in a GB200 NVL72 rack (4.56 times less). Part of that gap is the NVL72 GPU's larger per-chip bandwidth; the rest is the bisection.

AMD Helios: a 72-GPU rack, double width. AMD's Helios reference rack holds 72 Instinct MI450 GPUs with "up to 260 TB/s of scale-up interconnect bandwidth", 3.611 TB/s per GPU if spread evenly, and 43 TB/s of Ethernet scale-out. Each GPU has up to 432 GB of HBM4 at 19.6 TB/s. It is built on the Open Rack Wide specification, a double-wide liquid-cooled rack submitted to OCP, and AMD frames it around open standards (OCP, UALink, Ultra Ethernet). AMD expects volume deployment in 2026, so it is marked planned.

AWS Trainium UltraServers. AWS's Trn2 UltraServer joins 64 Trainium2 chips across four 16-chip instances with NeuronLink, its own chip-to-chip interconnect. The Trainium3 UltraServer scales to 144 chips with NeuronSwitch, "an all-to-all fabric": like NVIDIA, a switch rather than direct links. AWS does not state per-chip interconnect bandwidth on the pages used.

Design	Accelerators	Scale-up	Per chip (both ways)	HBM per chip
Google TPU v5p podproduct	8,960[tpu-v5p]	3D torus; 4x4x4 cubes joined by optical circuit switches[tpu-v5p]	1.2 TB/s[tpu-v5p]	2.765 TB/s[tpu-v5p]
Google TPU7x (Ironwood) podproduct	9,216[tpu-7x]	not disclosed[tpu-7x]	1.2 TB/s[tpu-7x]	7.38 TB/s[tpu-7x]
AMD Helios (72 x Instinct MI450)planned	72[helios-blog]	not disclosed[helios-blog]	3.611 TB/s (derived: total ÷ chips)	19.6 TB/s[helios-blog]
AWS Trn2 UltraServer (64 x Trainium2)product	64[trn2-page]	NeuronLink across four 16-chip instances[trn2-page]	not disclosed[trn2-page]	—
AWS Trainium3 UltraServer (144 x Trainium3)product	144[trainium-page]	NeuronSwitch all-to-all fabric[trainium-page]	not disclosed[trainium-page]	4.9 TB/s[trainium-page]

What each design changes about the ladder of chapter 1 is mostly the size and shape of the fast domain: 64 to 144 chips behind switches, 72 GPUs in a rack, or thousands of chips in a reconfigurable torus with more hops between them. The checks below confirm the per-chip and total figures in the vendors' own pages agree with each other.

Check	Kind	Model	Published	Ratio
AMD Helios total HBM (B): 72 x per-GPU [helios-blog]	spec cross-check	3.1104e13	3.1000e13	1.0034
AMD Helios total HBM bandwidth (B/s): 72 x per-GPU [helios-blog]	spec cross-check	1.4112e15	1.4000e15	1.0080

Check	Kind	Model	Published	Ratio
Trn2 UltraServer FP8 (FLOP/s): 4 x a 16-chip instance [trn2-page]	spec cross-check	8.3200e16	8.3200e16	1.0000
Trn2 UltraServer HBM bandwidth (B/s): 4 x a 16-chip instance [trn2-page]	spec cross-check	1.8400e14	1.8500e14	0.9946
Trainium3 UltraServer HBM (B): 144 x per-chip [trainium-page]	spec cross-check	2.0736e13	2.0700e13	1.0017
Trainium3 UltraServer HBM bandwidth (B/s): 144 x per-chip [trainium-page]	spec cross-check	7.0560e14	7.0600e14	0.9994

Maths

On a ring of $k$ nodes ( $k$ even) the distances from one node to all $k$ sum to $k^2/4$ . In $n$ dimensions a node's coordinates add independently, so the distances to all $N = k^n$ nodes sum to $n\,k^{n-1}\cdot k^2/4$ :

\overline{h} = \frac{n\,k^{n+1}}{4\,(N-1)}, \qquad h_{\max} = n\,\frac{k}{2}, \qquad L_{\text{bisection}} = 2\,k^{\,n-1}.

In an all-to-all, each chip sends $S/N$ to every other; the $N/2$ chips on one side send $(N/2)(N/2)(S/N) = NS/4$ bytes across the cut, each way, over $L_{\text{bisection}}$ links of bandwidth $B_{\text{link}}$ .

Code

A chip's hop distance from chip 0, cut from src/lib/racks/model.ts:

for (let i = 0; i < n; i++) {
  const c = x % k;
  d += Math.min(c, k - c);
  x = Math.floor(x / k);
}