64 GPU AI cluster network: GB300 NVL72
You cannot buy 64 GPUs of NVL72: the rack is the unit, so this rounds up to 72. Those 72 links at 800G fit one Quantum-X800, and a lone rack is already one accelerator over NVLink anyway.
- GPUs72
- Racks of GPUs1
- Compute switches1
- Fabric tiers1
- Total racks2
- Total draw129 kW
The build
| Platform | NVIDIA GB300 NVL72, 72 × GB300 (Blackwell Ultra) per rack |
|---|---|
| Compute fabric | infiniband, 72 rails at 800G |
| Leaf | 1 × Q3400-RA (Quantum-X800) |
| Spine | not required at this scale |
| Core | not required at this scale |
| Racks | 1 compute + 1 network, 1 rack per rack, limited by fixed |
| Power | 120 kW compute + 9 kW network = 129 kW |
How this fabric was sized.
72 endpoints at 800G fit a single Q3400-RA (Quantum-X800) with 72 ports spare. No spine tier is needed and rail optimization buys nothing here: every GPU is already one hop from every other. A rail-optimized arrangement at this size would cost one leaf per rail plus spines to join them, for no gain. Note this is a single point of failure for the compute fabric; a second switch for redundancy is a resilience decision rather than a capacity one.
Asked for 64, built 72. An NVL72 rack is indivisible, so GPU counts round to multiples of 72. 1 racks is the smallest build that meets the target.
Fabrics
Compute
72 endpoints across 72 rails, 1 tier, realised oversubscription 1:1 (non-blocking).
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | NVIDIA Q3400-RA (Quantum-X800) | 1 | 144 | 0 | 72 / 144 |
Storage
50 endpoints across 1 rail, 2 tiers, realised oversubscription 1.91:1.
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | NVIDIA QM9700 (Quantum-2) | 2 | 42 | 22 | 94 / 128 |
| spine | NVIDIA QM9790 (Quantum-2, externally managed) | 1 | 64 | 0 | 44 / 64 |
In-band management
18 endpoints across 1 rail, 2 tiers, realised oversubscription 3.67:1.
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | Cisco Nexus 93600CD-GX | 1 | 22 | 6 | 24 / 28 |
| spine | Cisco Nexus 93600CD-GX | 1 | 28 | 0 | 6 / 28 |
Out-of-band management
29 endpoints across 1 rail, 1 tier, realised oversubscription 1:1 (non-blocking).
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | Cisco Nexus 9348GC-FX3 | 1 | 48 | 4 | 29 / 48 |
Bill of materials
Every line carries the calculation that produced its quantity. Prices are deliberately absent: the catalog ships without list prices, because inventing them would be worse than leaving them out.
| Item | Qty | Why this quantity |
|---|---|---|
| Servers | ||
| GB300 NVL72 (72 x GB300 (Blackwell Ultra)) | 1 | 64 GPUs requested / 72 GPUs per node = 1 nodes (72 GPUs). |
| Network adapters | ||
NVIDIA ConnectX-8 SuperNIC C8180 (XDR 800G)900-9X81E-00EX-ST0 |
72 | 1 nodes x 72 compute NICs per node (one per rail). |
| Compute fabric switches | ||
NVIDIA Q3400-RA (Quantum-X800)920-9B36F-00RX-8S0 |
1 | 1 across Compute fabric (single switch) leaf. |
NVIDIA QM9700 (Quantum-2)MQM9700-NS2F |
2 | 2 across Storage fabric leaf. |
NVIDIA QM9790 (Quantum-2, externally managed)MQM9790-NS2F |
1 | 1 across Storage fabric spine. |
| Management switches | ||
Cisco Nexus 9348GC-FX3N9K-C9348GC-FX3 |
1 | 1 across Out-of-band management leaf. |
Cisco Nexus 93600CD-GXN9K-C93600CD-GX |
2 | 2 across In-band management leaf, In-band management spine. |
| Optics — transceivers | ||
800G OSFP 2xSR4 twin-port (NDR)MMA4Z00-NS |
46 | 44 x 400G Storage leaf to Storage spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP 2xSR4 twin-port (NDR) at Storage leaf and 800G OSFP 2xSR4 twin-port (NDR) at Storage spine, over MPO-12 APC OM4 multi-mode trunk. The Storage leaf module is twin-port, so it terminates 2 links and only 0.5 module is needed per link. |
| Cables and assemblies | ||
| 100G QSFP28 passive DAC | 19 | 18 x 100G In-band NIC to In-band switch at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed. |
800G OSFP passive DACMCP4Y10-N00A |
75 | 72 x 800G Compute NIC to Leaf at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed. |
800G OSFP to 2x400G OSFP splitter DACMCP7Y00-N001 |
26 | 18 x 400G Storage NIC to Storage leaf at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed; one cage breaks out to 2 links. |
| Cat6a patch lead | 30 | 29 x 1G BMC / device management port to OOB switch at 2 m. 1G over structured copper, 2 m within the 100 m limit for Cat6a patch lead. |
| Structured fibre | ||
| MPO-12 APC OM4 multi-mode trunk | 46 | 44 x 400G Storage leaf to Storage spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP 2xSR4 twin-port (NDR) at Storage leaf and 800G OSFP 2xSR4 twin-port (NDR) at Storage spine, over MPO-12 APC OM4 multi-mode trunk. The Storage leaf module is twin-port, so it terminates 2 links and only 0.5 module is needed per link. |
| Racks | ||
| Rack | 2 | 1 compute racks at 1 nodes each (fixed-limited) plus 1 network racks. |
| Power | ||
| Rack PDU | 4 | 2 racks x 2 PDUs per rack. |