Rail-optimized network design
The topology behind every large GPU cluster, and the sizing mistake that under-orders switches.
In a rail-optimized fabric, NIC i of every GPU node connects to leaf switches serving rail i, and only rail i. An eight-GPU node with one NIC per GPU has eight rails, and its eight NICs land on eight different leaf switches.
The payoff is one switch hop. Traffic between the same GPU position on different nodes never climbs to the spine tier and back down, and that is most of what collective operations like all-reduce generate.
Why each rail is sized on its own
A leaf switch never mixes rails. Everything about the sizing follows from that one constraint: each rail rounds up to a whole number of leaf switches on its own, and you multiply by the rail count afterwards.
The tempting shortcut is to total the endpoints, divide by the leaf port count, and round once. It gives the wrong answer, and it errs towards ordering too little hardware.
Worked example: 40 nodes, 8 rails, 64-port leaf switches.
Flat reading: 40 × 8 = 320 endpoints, 320 ÷ 64 = 5 leaf switches.
Rail-optimized: each rail has 40 endpoints, and 40 ÷ 64 rounds up to 1 leaf. Eight rails × 1 = 8 leaf switches.
The flat method is three switches short. The gap widens with every rail whose endpoint count sits just above a multiple of the port count.
Uplinks and oversubscription
Leaf ports split between downlinks to nodes and uplinks to the spine tier. A 1:1 non-blocking design gives them equal counts, so a 64-port switch becomes 32 down and 32 up. Loosening that to 2:1 buys downlink capacity at the cost of bisection bandwidth. For AI training that is usually a bad trade, since collectives are the workload that fills the bisection.
Round uplink counts up, never down. Rounding down delivers worse oversubscription than was asked for, and nothing in the parts list shows it. It surfaces months later as a bandwidth ceiling nobody can account for.
Rail-isolated planes against a shared spine
Above the leaf tier there is a real choice. A rail-isolated design gives each rail its own spine switches, so a rail is an independent fabric top to bottom. A shared spine pools the uplinks from every rail across one common tier.
The two produce materially different switch counts and neither is an obvious default, so make it an explicit decision. Rail isolation gives cleaner fault domains and simpler routing. A shared spine uses spine ports more efficiently when the rail count is low.
Where it stops working
Rail-optimized design assumes one NIC per GPU, with the NIC count matching the rail count. Two common systems break that. A PCIe server whose rail count is bounded by free slots is not rail-optimized in the same sense, and a rack-scale NVL72 system has a separate internal fabric this analysis does not cover.
Related
- Fat-tree sizing and tier boundaries — how far two tiers stretch before a third becomes unavoidable.
- DGX SuperPOD switch counts — the published figures this maths reproduces.
- InfiniBand against Spectrum-X Ethernet — how the fabric choice changes the link count.