Rail-optimized sizing that rounds per rail
A leaf switch never mixes rails, so each rail rounds up independently. Size the fabric as one flat pool and you under-count leaf switches. On a 40-node cluster that alone costs you six.
NVIDIA rail-optimized fabrics from 128 to 20,000 GPUs, across InfiniBand, Spectrum-X Ethernet, and Cisco Nexus. Describe your servers, answer a handful of questions, and get switch, optic, cable, rack, and power counts you can audit line by line.
A leaf switch never mixes rails, so each rail rounds up independently. Size the fabric as one flat pool and you under-count leaf switches. On a 40-node cluster that alone costs you six.
A 64-port switch caps a two-tier fabric at 2,048 endpoints per rail. The designer works out where that boundary falls for your setup instead of guessing, and says when a third tier becomes unavoidable.
Form factor, vendor coding, reach, and fiber connector all have to agree. OSFP will not fit QSFP-DD, and MPO-12 will not patch to MPO-16. Incompatible combinations are refused with the reason, not quietly substituted.
Storage, in-band, and out-of-band management are sized too. The OOB network alone needs a port for every node, every switch, and every PDU. That runs to thousands at scale, and it is the line most often missing from a quote.
AI racks fill their power budget long before their space. A 10 kW node in a 45 kW rack means four per rack with 10U left stranded. The output says which constraint bound it.
Every line carries the calculation that produced its quantity. Specs not confirmed against a vendor datasheet are badged, so nothing unverified reaches a customer quote by accident.
The topology maths is checked against published NVIDIA DGX SuperPOD reference architectures, not just asserted. A single scalable unit reproduces the documented 8 leaf and 4 spine switches; four scalable units reproduce 32 leaf and 16 spine. Tests cover those figures on every change.
In a rail-optimized design, NIC i of every GPU node connects to leaf switches serving rail i. A leaf switch never mixes rails, so traffic between the same GPU position on different nodes crosses a single hop. For sizing, that means each rail rounds up to a whole number of leaf switches on its own, and you multiply by the rail count afterwards. Treat the fabric as one flat pool of endpoints and you under-count the leaf tier.
A non-blocking two-tier fat tree of P-port switches supports P²/2 endpoints per rail, and a three-tier fat tree supports P³/4. A 64-port switch therefore caps a two-tier rail at 2,048 endpoints and a three-tier one at 65,536. Each rail is sized against the node count, not the total GPU count, so the tier boundary sits much higher than a flat reading suggests. Switches with breakout ports raise it further.
Both are used in production AI clusters and NVIDIA publishes reference architectures for each. The difference that shows up in a bill of materials is the port arrangement: ConnectX-8 runs one 800G port in InfiniBand mode but two 400G ports in Ethernet mode, so an Ethernet build of the same servers terminates twice as many links at half the speed. Aggregate bandwidth is unchanged; switch, optic and cable counts are not. Ethernet designs also commonly use multiple planes, which changes the topology again.
Yes. An NVL72 rack carries three distinct networks. NVLink joins the 72 GPUs inside the rack, each GPU having 18 fifth-generation links to the in-rack NVSwitches over a copper backplane. It ships with the rack, needs no optics, and is not a purchasing decision. Scale-out east and west traffic between racks runs over ConnectX SuperNICs, one per GPU. North and south storage and management traffic runs over BlueField-3 DPUs. You buy switches and optics for the last two only.
Six things have to agree before a link works. Form factor, because OSFP will not fit a QSFP-DD cage. Vendor coding. Reach. Fibre type. Fibre connector, because MPO-12 will not patch to MPO-16. And ferrule polish, because APC will not mate with UPC. Heatsink form is a seventh consideration, since a finned-top module will not fit a switch cage designed for flat-top optics. Get any one of them wrong and the link will not come up.
Four fabrics, not one. The compute fabric carries GPU to GPU traffic and gets most of the attention. The storage fabric, in-band management network and out-of-band management network are all separately cabled. Out-of-band is the one most often missing from a quote. It needs a port for every node BMC, every switch in every other fabric, and every PDU, which runs to thousands of ports at scale. On top of that sit optics and cables for every link, racks, and the power distribution to fill them.