The package

Compute dies and HBM stacks on one package: why memory bandwidth and capacity set the shape, and how many GPUs a model's weights need.

Loading the animation…

Concept

A data-centre GPU is a package: one or two compute dies in the middle and stacks of HBM (high-bandwidth memory) around them, on a silicon interposer or bridge that carries thousands of wires between them. Each HBM stack is DRAM dies stacked on a base die, with a 1,024-bit interface in HBM3 and HBM3E and a 2,048-bit one in HBM4 ("double from the previous generation", SK hynix), which is much of why Rubin's per-stack bandwidth jumps; that width is why the stacks sit millimetres from the compute die instead of on the board.

The two packages in the animation:

H100 SXM5: one die and 5 HBM3 stacks of 16 GB, 670 GB/s each. The full GH100 die has six HBM sites; the SXM5 product uses five (NVIDIA's Hopper in-depth blog).
Blackwell Ultra: two reticle-limited dies joined by a 10 TB/s die-to-die link into one GPU, and 8 twelve-high HBM3E stacks of 36 GB, 1 TB/s each: 288 GB in all.

Capacity decides how many GPUs a model needs before any parallelism choice is made. Llama 3.1 405B in FP8 is 405 GB of weights: 6 H100s, or 2 Blackwell Ultras.

Bandwidth decides how fast those weights can be used. Generating one token at batch size 1 reads every weight once, so a GPU holding 203 GB of them cannot produce a token faster than 25.3 ms, whatever its arithmetic. Batching many requests shares that read; the Serving site's roofline chapter shows where it stops helping.

Arithmetic has grown faster than memory bandwidth. Dense FP8 per byte of HBM bandwidth (the roofline's ridge point) is 591 flop/byte on H100, 625 on Blackwell Ultra and 911 on Rubin. And capacity has grown faster than bandwidth on Blackwell Ultra: reading its whole HBM once takes 36 ms, against 23.9 ms on H100; Rubin's HBM4 brings it to 15 ms.

Maths

With $n$ stacks the package's capacity and bandwidth are $C = n\,c$ and $B = n\,b$ for one stack's $c$ and $b$ . A model of $P$ parameters at $\beta$ bytes each has $W = \beta P$ bytes of weights, so

N_{\text{GPU}} = \left\lceil \frac{\beta P}{C} \right\rceil, \qquad t_{\text{read}} = \frac{\beta P / N_{\text{GPU}}}{B}, \qquad \text{tokens/s at batch 1} \le \frac{1}{t_{\text{read}}} .

The ridge point is $F / B$ for dense peak $F$ : a kernel doing fewer than $F/B$ operations per byte it reads from HBM waits on memory.

Code

The fit, cut from src/lib/racks/model.ts:

const wb = weightsBytes(mid, fmt);
const n = Math.ceil(wb / g.hbm_bytes);
const share = wb / n;
return {
  weights: wb,
  gpus: n,
  share,
  fill: share / g.hbm_bytes,
  read_time: share / g.hbm_bw,
  tokens_per_s: g.hbm_bw / share,
};

Five generations of package

	H100 SXM5	H200 SXM	Blackwell (GB200 NVL72)	Blackwell Ultra (GB300 NVL72)	Rubin (Vera Rubin NVL72)preliminary
HBM capacity	80 GB[h100-page]	141 GB[h200-page]	186 GB (derived)[gb200-page]	288 GB[bwu-blog]	288 GB[vr-page]
HBM bandwidth	3.35 TB/s[h100-page]	4.8 TB/s[h200-page]	8 TB/s (derived)[gb200-page]	8 TB/s[bwu-blog]	19.2 TB/s[vr-page]
HBM stacks	5[hopper-blog]	not disclosed[h200-page]	not disclosed[bwu-blog]	8[bwu-blog]	not disclosed[vr-page]
hbm_gen	HBM3[hopper-blog]	HBM3E[bwu-blog]	HBM3E[gb200-page]	HBM3E[bwu-blog]	HBM4[vr-page]
Compute dies	1[bwu-blog]	1[bwu-blog]	2[bw-arch]	2[bwu-blog]	not disclosed[vr-page]
Transistors	80 billion[bwu-blog]	80 billion[bwu-blog]	208 billion[bw-arch]	208 billion[bwu-blog]	not disclosed[vr-page]
FP8, dense	1,980 TFLOP/s (derived)[h100-page]	1,980 TFLOP/s (derived)[h200-page]	5,000 TFLOP/s[bwu-blog]	5,000 TFLOP/s[bwu-blog]	17,500 TFLOP/s[vr-page]
Max power	700 W[h100-page]	700 W[h200-page]	1.2 kW[bwu-blog]	1.4 kW[bwu-blog]	not disclosed[vr-page]

Blackwell's two dies are each as large as the lithography reticle allows; joining two of them is how the GPU grew past one die's limit. Rubin's specifications come from NVIDIA's Vera Rubin NVL72 page, whose NVLink figures NVIDIA marks preliminary. Each Blackwell row is the GPU as it sits in its NVL72 rack: per-GPU memory is the superchip's figure split two ways (186 GB for GB200), dense FP8 is NVIDIA's per-GPU table figure and power is the maximum board power (TGP). AI Accelerators Explained uses the same B200. The air-cooled HGX B200 is a different board, rated lower (4.5 PFLOPS dense FP8 per GPU, from its page).