Cost per token

Capital, electricity and utilisation turned into cost per million tokens and tokens per megawatt, with every price marked illustrative.

Loading the animation…

Concept

The cost of a token has two big parts. Capital: the rack, its share of the network and the building, written off over its useful life. Energy: the rack's power, plus the facility's overhead for cooling and conversion (the PUE, power usage effectiveness: facility power over IT power), times the price of electricity. Divide by the tokens the rack serves, which depends on how fast it serves them and how much of the time it is busy.

Every price here is illustrative. NVIDIA does not publish rack prices, electricity prices vary several-fold between sites, and utilisation is an operator's secret. The chapter uses round numbers, labelled, to show the shape of the sum, not its value: $3.5M a rack written off over 4 years, PUE 1.2, $0.1 per kWh and 70% utilisation. Two inputs are not invented: the rack draws its stated 120 kW (NVIDIA's GB200 NVL72 figure, taken as its draw all year), and serves the 52,824 output tokens a second that chapter 8's simulated rack sustained within its latency targets.

With those inputs a million output tokens costs $0.858, and 87% of it is capital: electricity is $126k a year against $875k of write-off. That is the usual shape for AI hardware today: the machine costs far more than its electricity, so idle hardware is the expensive kind. Cheap capital, cheap power and high utilisation give $0.375; dear capital, dear power and half-idle hardware give $1.73: a dearer rack's cost spread over only 834 billion tokens a year.

Tokens per megawatt. On the same basis (rack at its stated power, times PUE) the rack serves 256,785 output tokens a second per megawatt of facility power, on average over the year. Chapter 8 quotes 850,011: a different basis, the simulator's GPU power alone while serving, with no idle time. Both are honest; a tokens-per-megawatt figure means little until its basis is stated.

What moves the number. Utilisation and throughput divide the capital, so everything in the earlier chapters that raises the tokens a rack serves within its targets (the right pool split, keeping the KV hand-off on NVLink, batching) lowers cost per token roughly in proportion. At 6:12, chapter 8's rack met its targets for only 25% of requests: the same capital, far fewer useful tokens. Energy matters most where power, not money, is the limit: a site with a fixed number of megawatts serves at most its tokens per megawatt times its megawatts.

Maths

With capital $C$ written off over $Y$ years, rack power $P$ , facility overhead PUE, electricity price $p_e$ per kWh, $h =$ 8,766 hours a year, a sustained output rate $R$ and utilisation $u$ :

\text{cost per token} = \frac{C/Y + (P/1000)\cdot \text{PUE}\cdot h \cdot p_e}{R\, u\, h \cdot 3600}, \qquad \frac{\text{tokens/s}}{\text{MW}} = \frac{R\,u}{P\cdot \text{PUE}/10^{6}}.

Code

The sum, cut from src/lib/racks/model.ts:

const energyKwh = (rackW / 1000) * pue * HOURS_PER_YEAR;
const energyUsd = energyKwh * usdPerKwh;
const capexYr = capex / years;
const tokensYr = tokS * util * HOURS_PER_YEAR * 3600;
const total = capexYr + energyUsd;