Disaggregated inference on a rack

Serving a large model has two phases with opposite needs. Prefill reads the whole prompt in one pass: thousands of tokens share each weight read, so it is limited by compute. Decode produces one token per request per step, re-reading the weights and the request's KV cache each time, so it is limited by memory bandwidth. Run both on the same GPUs and each gets in the other's way: a long prefill stalls every decode stream batched with it. Disaggregated serving gives each phase its own pool of GPUs and moves each request's KV cache from the prefill pool to the decode pool after its first token (DistServe and Splitwise, both 2023-24 papers; Serving, chapter 11 explains why).

A rack is a natural unit for it. Inside one NVL72-class rack every GPU is one NVLink switch hop from every other, so the KV hand-off never touches the network.

The simulated rack. The animation runs Disaggregated_Inference_Sim, a discrete-event simulator with a published validation section, in your browser: its own JavaScript engine, vendored byte for byte at commit ba37774 and checked here to reproduce its Python package bit for bit. The rack is 72 GPUs of the simulator's GB200 preset, one Blackwell GPU of a GB200 superchip (186 GB and 8 TB/s each, NVIDIA's per-GPU figures) as 18 instances of 4, tensor-parallel over NVLink, serving Llama 3 70B. 1,500 requests arrive at 90 a second, with prompts of 3,000 tokens and outputs of 600 on average. The SLOs: first token within 2 s, then 50 ms a token at most.

Sizing the pools. That load asks for 268,708 prompt tokens a second of prefill and 52,824 output tokens a second of decode. With 6 prefill and 12 decode instances, prefill cannot keep up: requests queue, TTFT reaches 6.9 s at the 99th percentile and only 25% of requests meet both SLOs, while the decode pool idles at 12.4 ms a token. Moving instances to prefill brings TTFT down to 420 ms at 12:6. Every instance taken from decode makes each decode instance batch more streams, so time per token rises: 16.8 ms at 12:6, 21 ms at 14:4. The right split follows the workload's prompt-to-output ratio, which is why disaggregated serving systems resize their pools as traffic changes.

The KV hand-off. Llama 3 70B's cache is 328 kB a token, so an average prompt hands over 983 MB. From a prefill instance's four GPUs that takes 273 µs over their NVLinks, 4.92 ms over their four 400 Gb/s network ports (the path between racks), and 78.6 ms over a single 100 GbE port. The simulator measures the same means, queueing included: 279 µs, 4.95 ms and 79.1 ms. The hand-off delays each request's second token, so it counts in time per output token (the simulator follows DistServe's definition); spread over 600 output tokens it costs little here: 17.2 ms against 16.8 ms at the 99th percentile. It matters more for long prompts with short answers, and when a pool's port is shared. Serving, chapter 12 and Inference Trade-offs, chapter 6 go further.

Context-optimised parts. If prefill is compute-bound, it does not need HBM. NVIDIA has announced Rubin CPX, a GPU with 128 GB of GDDR7 instead of HBM for "massive-context" prefill, expected at the end of 2026 (planned, not shipping). The simulation's context-optimised part is illustrative, not CPX: the rack GPU's own compute (the GB200 preset's), so that it differs from the rack's GPUs in memory only, with 128 GB of memory at 2 TB/s, a GDDR7-class bandwidth chosen by us, since NVIDIA states none. Put it in the prefill pool and TTFT barely moves (419 ms against 420 ms). Put it in the decode pool and time per token rises to 65.3 ms: only 13% of requests meet the SLOs. That asymmetry is the case for building different hardware for each pool (Inference Trade-offs, chapter 5).

The software that schedules it. Production systems route each request to a prefill worker, move its cache, and hand it to a decode worker, while watching both pools' queues. NVIDIA's open-source Dynamo describes itself as "the orchestration layer above inference engines" (vLLM, SGLang, TensorRT-LLM), with disaggregated serving and KV-aware routing among its features. The simulator's scheduler is far simpler: first come, first served in each pool, no routing by cache contents.

Energy. With the simulator's illustrative power coefficients, the rack spends 1.18 J per output token at 12:6, against 1.36 J at 6:12, where a backlog stretches the run out and idle power adds up (chapter 6 on power, chapter 11 on cost).