Cooling

Why air runs out at rack scale: cold plates, the coolant loop, CDUs and facility water, sized with Q = ṁ c_p ΔT.

Loading the animation…

Concept

Every watt a rack draws ends up as heat, and something has to carry it away. A fluid carries heat $Q = \dot m\,c_p\,\Delta T$ : its mass flow, times its specific heat, times how much it warms. Water's specific heat is 4,184 J/(kg K) (NIST's equation at 25 °C); air's is 1,001 J/(kg K), and a cubic metre of air weighs only 1.23 kg. So, volume for volume, water carries thousands of times more heat.

Why air runs out. To carry a 120 kW rack's heat, warming by 15 K, air needs 6.5 m³ a second (13,779 cubic feet a minute) through one rack. Water warming by 10 K needs 172 litres a minute: 2,267 times less volume. NVIDIA's own comparison is that data centres "once operated at 20 kW per rack" and that hyperscale facilities now support "over 135 kW per rack"; air was designed for the first.

The liquid loop. In a GB200 NVL72, the CPUs and GPUs sit under cold plates, and coolant runs up and down the rack through manifolds; networking and storage stay air-cooled (NVIDIA's DGX GB200 user guide). The animation follows the coolant:

the coolant distribution unit (CDU) sends it out a few kelvin above the facility water that cools it (the approach; 5 K for the partner CDU NVIDIA cites, rated at 2 MW);
the supply manifold feeds the compute trays in parallel;
the cold plates take the heat, warming the coolant by $\Delta T$ ;
the return manifold brings it back to the CDU, whose exchanger hands the heat to the facility water, which leaves for chillers, cooling towers or dry coolers.

With 90% of the heat on cold plates (an illustrative share: NVIDIA does not publish one), facility water at 30 °C and a 10 K rise, the loop runs at 155 L/min, 0.143 kg/s through each tray, from 35 °C to 45 °C. One 2 MW CDU serves 18 such racks.

Warm water. The warmer the coolant can run, the less work the facility does to cool it: with warm enough facility water, dry coolers that use outside air can replace chillers. NVIDIA's 100 MW Vera Rubin reference factory states a 45 °C inlet temperature. Try 40 °C facility water in the animation: the coolant then leaves the CDU at 45 °C. The limit is the chips: the cold plate has to keep each die below its own maximum while the coolant arrives warmer.

One megawatt. At 1 MW (NVIDIA's planned Kyber-class rack), a 10 K rise needs 1,434 L/min of water, and air would need 54 m³ a second: liquid is not optional.

Maths

The coolant's mass flow, and the volume of air that would carry the same heat:

\dot m = \frac{Q_{\text{liquid}}}{c_{p,\text{water}}\,\Delta T_{\text{water}}}, \qquad \dot V_{\text{air}} = \frac{Q}{\rho_{\text{air}}\,c_{p,\text{air}}\,\Delta T_{\text{air}}}.

Water's $c_p$ comes from NIST's Shomate equation for liquid water at $T = 298.15$ K ( $t = T/1000$ , coefficients $A \ldots E$ , molar mass $M$ ):

c_p = \frac{A + Bt + Ct^2 + Dt^3 + E/t^2}{M}, \qquad c_{p,\text{air}} = \frac{\gamma R}{\gamma - 1}

with NASA's standard-day $\gamma =$ 1.4 and $R =$ 286 J/(kg K). The coolant leaves the CDU at the facility temperature plus the approach, and returns $\Delta T$ warmer.

Code

The rack's loop, cut from src/lib/racks/model.ts:

const qLiq = qW * liquidFrac;
const tSup = tFac + fnum(c, "cdu_approach_k");
const tRet = tSup + dt;
const kgS = qLiq / (waterCp() * dt);