Training clusters

How a training job's tensor, pipeline and data parallelism land on NVLink domains and the network, and how often a large job is interrupted.

Loading the animation…

Concept

A model too large for one GPU is split three ways (Training, chapters 11 and 12 explain each). Tensor parallelism splits every layer across a few GPUs, which then all-reduce their partial results twice per layer, every micro-batch. Pipeline parallelism gives each group of layers to a different set of GPUs and passes activations from stage to stage. Data parallelism runs copies of the whole arrangement on different data and averages their gradients once a step, which can overlap with the backward pass. The traffic falls in that order. Narayanan et al. (2021) give the rule for servers of $g$ GPUs: tensor parallelism up to degree $g$ , inside the server's NVLink domain, then pipeline parallelism across servers; data parallelism, the lightest, goes outermost.

The animation places a small job, one tensor-parallel group per frame, in the order Llama 3's training used: tensor-parallel ranks innermost, then pipeline stage, then replica. Each line joins a group to the stage before it: green when the activations stay on NVLink, dashed blue when they cross the network.

Llama 3 405B, re-housed. One stage of Llama 3 405B's pre-training ran on 16,384 H100 GPUs as TP 8 × PP 16 × DP 128, eight GPUs to an NVLink server. On those servers, all 2,048 tensor-parallel groups fit inside one server, and every one of the 1,920 pipeline hops crosses the network (1,920). Laid out the same way on NVL72 racks, the job fills 228 racks and only 213 of the pipeline hops cross a rack boundary; the rest move to NVLink. The data-parallel gradient averaging crosses the network either way (128 of 128 groups). This is an illustration of the layout rule, not a description of any real deployment.

Interruptions. At this scale, failure is routine. Over a 54-day period of Llama 3 405B pre-training Meta counted 419 unexpected interruptions; about 78% were confirmed or suspected hardware issues, and GPU issues alone were 58.7% (the paper's Table 5). In synchronous training one failed GPU stops the whole job, which restarts from its last checkpoint. As one rate, that is 0.00002 interruptions per GPU-hour, so a job of 16,384 GPUs is interrupted every 11,100 s on average. Young's rule (1974) sets the checkpoint interval at $\sqrt{2\,\delta M}$ for a save time $\delta$ and mean time between failures $M$ : with a 60 s save, every 1,160 s, losing about 10% of the time. At 100,000 GPUs that rises to 26%, which is why fast checkpointing, quick restart and automatic diagnosis matter as much as the GPUs (Training, chapter 15).

GPUs	Interruptions a day	Mean time between	Checkpoint every	Time lost
1,024	0.485	178,000 s	4,620 s	2.6%
4,096	1.94	44,500 s	2,310 s	5.2%
16,384	7.76	11,100 s	1,160 s	10.4%
100,000	47.4	1,820 s	468 s	25.6%

Maths

With $N$ GPUs failing independently at $\lambda$ per GPU-hour, the job's mean time between interruptions is $M = 1/(\lambda N)$ . A checkpoint every $\tau$ that takes $\delta$ to write loses, to first order, $\delta/\tau$ to saving and $\tau/2M$ to recomputing the work since the last checkpoint:

\text{lost} \approx \frac{\delta}{\tau} + \frac{\tau}{2M}, \qquad \frac{d\,\text{lost}}{d\tau} = 0 \;\Rightarrow\; \tau^{*} = \sqrt{2\,\delta M}.

From the Llama 3 record, $\lambda = 419 / (16{,}384 \times 54 \times 24)$ per GPU-hour.

Code

Young's interval, cut from src/lib/racks/model.ts:

const lam = failureRate() / 3600;
const mtbf = 1 / (lam * nGpus);
const tau = Math.sqrt(2 * saveS * mtbf);

Go deeper

Training across data centres, where the WAN is the slow link: the Compute in Transit results site measures gradient compression (DeMo's top-k against a fixed band an optical link module could compute), simulates when computing it in the network pays, and finds the case narrow.
Companion sites: Training, data parallelism, model parallelism and slow links; this site's chapter 4 (all-reduce inside the rack) and chapter 5 (the fabric between racks).
NVIDIA GPU series: SuperPOD, the hyperscale cluster blueprint, SHARP in-network reductions and NCCL and the all-reduce path.
Grattafiori et al., The Llama 3 Herd of Models (Table 4, section 3.3.4); Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM; Young, A first order approximation to the optimum checkpoint interval.