AI Racks Explained
The machines underneath, chip to data centre.
An interactive guide to the machines underneath AI: packages, NVLink racks, networks, power and cooling, with every hardware figure sourced and dated.
- Package1 GPUHBM 8 TB/s
- Compute tray4 GPUsNVLink 900 GB/s
- Rack72 GPUsNVLink 900 GB/s
- Scalable unit576 GPUsInfiniBand 50 GB/s
- SuperPOD9,216 GPUsInfiniBand 50 GB/s
- Hall59,976 GPUsInfiniBand 50 GB/s
How this is checkedEvery hardware figure carries its source and the date it was read. The model is cross-checked against published rack, switch and fabric totals it does not take as inputs, and its collectives against a measured NCCL all-reduce. How it is tested →
What you'll learn
A large model is trained and served by thousands of GPUs, and the way they are packaged, wired, powered and cooled decides what is fast. Inside one package a GPU reads its HBM at 8 TB/s. Inside an NVLink rack it reaches any of 72 GPUs at 900 GB/s per direction. One rung further out, over the network, it has 50 GB/s: a drop of 18 times. Most of the design of a training or inference cluster is about which traffic crosses which boundary.
Each chapter is built around an animation whose every frame comes from a small model tested against a Python reference. Every hardware figure carries its source and the date it was read, products whose specifications are preliminary or planned are marked, and the model is checked against published figures (its checks, labelled by what each one tests).
The centrepiece
Chapter 8 serves 1,500 requests on one 72-GPU rack with prefill and decode in separate pools, running a tested discrete-event simulator in your browser: split the rack, change the link the KV cache crosses, or put a cheaper context-optimised part in one pool, and watch latency and energy per token move.
Every hardware figure the model uses, with its source and access date: the specifications.
Built by a hardware engineer
The author designs hardware: the RTL, SoC and signal-integrity work behind this track is public too.
- A transformer decoder in RTL: SystemVerilog with 84 passing tests (a Python golden model and cocotb), synthesised in Vivado 2025.2 for an Artix-7: its 4 × 4 systolic array is 455 LUTs, 901 flip-flops and 16 DSPs at 156.8 MHz (the README's synthesis table).
- The decoder, module by module: A walkthrough of every RTL module, the verification and the synthesis numbers.
- RTL challenges for LLM accelerators: Systolic array, mixed-precision MAC, softmax pipeline, KV cache and more, each simulated (two run inside Systolic Arrays Explained).
- FPGA design challenges: FPGA architecture, synthesis, timing closure and the Vivado and Quartus flows, with RTL exercises.
- A five-stage RISC-V SoC: BRV32P: a pipelined RV32IMC core with caches and an AXI4-Lite fabric in Verilog, and a MESI cache controller (Cache_Controller_MESI).
- Modern SoC design: Slide decks from chiplets and advanced packaging to on-chip interconnects and silicon implementation.
- Equalisation in high-speed serial links: One 28 GBd backplane channel end to end: S-parameters, CTLE, FFE and DFE, a noise budget.
- Signal integrity: Seventeen decks on getting a signal from one chip to another intact, every number computed by a model.
- All the hardware decks: RISC-V, Arm, AMBA, simulators, photonics and more.
Part of the companion sites' Hardware track, with GPU Kernels Explained (how one GPU executes the maths) and Systolic Arrays Explained (inside a matrix engine). What runs on these machines: Training and Serving. How this site was built, and how to check it: about.