512 GPU AI cluster network: DGX B300 — Spectrum-X
Two independent planes, 2x400GbE per GPU. ConnectX-8 cannot run 1x800G in Ethernet mode, so this terminates twice the links of the InfiniBand build.
- GPUs512
- Nodes64
- Compute switches24
- Fabric tiers2
- Total racks34
- Total draw866 kW
The build
| Platform | NVIDIA DGX B300 (Spectrum-X Ethernet), 8 × B300 (Blackwell Ultra) per node |
|---|---|
| Compute fabric | ethernet, 16 rails at 400G |
| Leaf | 16 × SN5600 (Spectrum-4) |
| Spine | 8 × SN5600 (Spectrum-4) |
| Core | not required at this scale |
| Racks | 32 compute + 2 network, 2 nodes per rack, limited by power |
| Power | 832 kW compute + 34 kW network = 866 kW |
Fabrics
Compute
1,024 endpoints across 16 rails, 2 tiers, realised oversubscription 1:1 (non-blocking).
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | NVIDIA SN5600 (Spectrum-4) | 16 | 64 | 64 | 2,048 / 2,048 |
| spine | NVIDIA SN5600 (Spectrum-4) | 8 | 64 | 0 | 1,024 / 1,024 |
Storage
160 endpoints across 1 rail, 2 tiers, realised oversubscription 1.91:1.
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | NVIDIA SN5600 (Spectrum-4) | 4 | 42 | 22 | 248 / 256 |
| spine | NVIDIA SN5600 (Spectrum-4) | 2 | 64 | 0 | 88 / 128 |
In-band management
128 endpoints across 1 rail, 2 tiers, realised oversubscription 3.67:1.
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | Cisco Nexus 93600CD-GX | 6 | 22 | 6 | 164 / 168 |
| spine | Cisco Nexus 93600CD-GX | 2 | 28 | 0 | 36 / 56 |
Out-of-band management
174 endpoints across 1 rail, 2 tiers, realised oversubscription 1:1 (non-blocking).
| Tier | Switch | Count | Down/sw | Up/sw | Ports used |
|---|---|---|---|---|---|
| leaf | Cisco Nexus 9348GC-FX3 | 4 | 48 | 4 | 174 / 192 |
| spine | Cisco Nexus 93180YC-FX3 | 2 | 48 | 0 | 16 / 96 |
Bill of materials
Every line carries the calculation that produced its quantity. Prices are deliberately absent: the catalog ships without list prices, because inventing them would be worse than leaving them out.
| Item | Qty | Why this quantity |
|---|---|---|
| Servers | ||
| DGX B300 — Spectrum-X (8 x B300 (Blackwell Ultra)) | 64 | 512 GPUs requested / 8 GPUs per node = 64 nodes (512 GPUs). |
| Network adapters | ||
NVIDIA ConnectX-8 SuperNIC C8240 (dual 400G)900-9X81Q-00CN-ST0 |
1,024 | 64 nodes x 16 compute NICs per node (one per rail). |
| Compute fabric switches | ||
NVIDIA SN5600 (Spectrum-4)920-9N42F-00RI-7C0 |
30 | 30 across Compute fabric (rail-optimized) leaf, Compute fabric (rail-optimized) spine, Storage fabric leaf, Storage fabric spine. |
| Management switches | ||
Cisco Nexus 93180YC-FX3N9K-C93180YC-FX3 |
2 | 2 across Out-of-band management spine. |
Cisco Nexus 9348GC-FX3N9K-C9348GC-FX3 |
4 | 4 across Out-of-band management leaf. |
Cisco Nexus 93600CD-GXN9K-C93600CD-GX |
8 | 8 across In-band management leaf, In-band management spine. |
| Optics — transceivers | ||
25G SFP28 SRSFP-25G-SR-S |
33 | 16 x 25G OOB switch to OOB aggregation at 30 m. 30 m run needs discrete optics over structured fiber. 25G SFP28 SR at OOB switch uplink and 25G SFP28 SR at OOB aggregation, over LC duplex OM4 multi-mode patch. |
800G OSFP 2xSR4 twin-port (NDR)MMA4Z00-NS |
1,055 | 1024 x 400G Leaf to Spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP 2xSR4 twin-port (NDR) at Compute leaf and 800G OSFP 2xSR4 twin-port (NDR) at Compute spine, over MPO-12 APC OM4 multi-mode trunk. The Compute leaf module is twin-port, so it terminates 2 links and only 0.5 module is needed per link. |
| 800G OSFP SR8 single-link (Spectrum-X) | 182 | 88 x 800G Storage leaf to Storage spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP SR8 single-link (Spectrum-X) at Storage leaf and 800G OSFP SR8 single-link (Spectrum-X) at Storage spine, over MPO-12 APC OM4 multi-mode trunk. |
| Cables and assemblies | ||
| 100G QSFP28 passive DAC | 132 | 128 x 100G In-band NIC to In-band switch at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed. |
800G OSFP to 2x400G QSFP112 splitter DACMCP7Y10-N001 |
610 | 1024 x 400G Compute NIC to Leaf at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed; one cage breaks out to 2 links. |
| Cat6a patch lead | 180 | 174 x 1G BMC / device management port to OOB switch at 2 m. 1G over structured copper, 2 m within the 100 m limit for Cat6a patch lead. |
| Structured fibre | ||
| LC duplex OM4 multi-mode patch | 17 | 16 x 25G OOB switch to OOB aggregation at 30 m. 30 m run needs discrete optics over structured fiber. 25G SFP28 SR at OOB switch uplink and 25G SFP28 SR at OOB aggregation, over LC duplex OM4 multi-mode patch. |
| MPO-12 APC OM4 multi-mode trunk | 1,146 | 1024 x 400G Leaf to Spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP 2xSR4 twin-port (NDR) at Compute leaf and 800G OSFP 2xSR4 twin-port (NDR) at Compute spine, over MPO-12 APC OM4 multi-mode trunk. The Compute leaf module is twin-port, so it terminates 2 links and only 0.5 module is needed per link. |
| Racks | ||
| Rack | 34 | 32 compute racks at 2 nodes each (power-limited) plus 2 network racks. |
| Power | ||
| Rack PDU | 68 | 34 racks x 2 PDUs per rack. |