AI Network Designer Design a network →

128 GPU AI cluster network: DGX B200

Cisco Nexus on Spectrum-4 silicon. Breaking 64 ports of 800G into 128 of 400G means all 128 endpoints land on a single switch.

The build

PlatformNVIDIA DGX B200, 8 × B200 per node
Compute fabricethernet, 8 rails at 400G
Leaf1 × Nexus 9164E-NS4-O (Spectrum-X)
Spinenot required at this scale
Corenot required at this scale
Racks6 compute + 1 network, 3 nodes per rack, limited by fixed
Power229 kW compute + 10 kW network = 239 kW

How this fabric was sized.

128 endpoints at 400G fit a single Nexus 9164E-NS4-O (Spectrum-X) with 0 ports spare. No spine tier is needed and rail optimization buys nothing here: every GPU is already one hop from every other. A rail-optimized arrangement at this size would cost one leaf per rail plus spines to join them, for no gain. Note this is a single point of failure for the compute fabric; a second switch for redundancy is a resilience decision rather than a capacity one.

Fabrics

Compute

128 endpoints across 8 rails, 1 tier, realised oversubscription 1:1 (non-blocking).

TierSwitchCountDown/swUp/swPorts used
leaf Cisco Nexus 9164E-NS4-O (Spectrum-X) 1 128 0 128 / 128

Storage

64 endpoints across 1 rail, 2 tiers, realised oversubscription 1.91:1.

TierSwitchCountDown/swUp/swPorts used
leaf Cisco Nexus 9164E-NS4-O (Spectrum-X) 2 42 22 108 / 128
spine Cisco Nexus 9164E-NS4-O (Spectrum-X) 1 64 0 44 / 64

In-band management

32 endpoints across 1 rail, 2 tiers, realised oversubscription 3.67:1.

TierSwitchCountDown/swUp/swPorts used
leaf Cisco Nexus 93600CD-GX 2 22 6 44 / 56
spine Cisco Nexus 93600CD-GX 1 28 0 12 / 28

Out-of-band management

38 endpoints across 1 rail, 1 tier, realised oversubscription 1:1 (non-blocking).

TierSwitchCountDown/swUp/swPorts used
leaf Cisco Nexus 9348GC-FX3 1 48 4 38 / 48

Bill of materials

Every line carries the calculation that produced its quantity. Prices are deliberately absent: the catalog ships without list prices, because inventing them would be worse than leaving them out.

ItemQtyWhy this quantity
Servers
DGX B200 (8 x B200) 16 128 GPUs requested / 8 GPUs per node = 16 nodes (128 GPUs).
Network adapters
NVIDIA ConnectX-7 400G (NDR)
MCX75310AAS-NEAT		 128 16 nodes x 8 compute NICs per node (one per rail).
Compute fabric switches
Cisco Nexus 9164E-NS4-O (Spectrum-X)
N9164E-NS4-O		 4 4 across Compute fabric (single switch) leaf, Storage fabric leaf, Storage fabric spine.
Management switches
Cisco Nexus 9348GC-FX3
N9K-C9348GC-FX3		 1 1 across Out-of-band management leaf.
Cisco Nexus 93600CD-GX
N9K-C93600CD-GX		 3 3 across In-band management leaf, In-band management spine.
Optics — transceivers
800G OSFP VR8 (multi-mode)
OSFP-800G-VR8		 91 44 x 800G Storage leaf to Storage spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP VR8 (multi-mode) at Storage leaf and 800G OSFP VR8 (multi-mode) at Storage spine, over MPO-12 APC OM4 multi-mode trunk.
Cables and assemblies
100G QSFP28 passive DAC 33 32 x 100G In-band NIC to In-band switch at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed.
800G OSFP to 2x400G OSFP splitter DAC
MCP7Y00-N001		 99 128 x 400G Compute NIC to Leaf at 2 m. 2 m run is within the 2 m passive copper limit. Optics are integrated into the assembly, so no separate transceivers are needed; one cage breaks out to 2 links.
Cat6a patch lead 40 38 x 1G BMC / device management port to OOB switch at 2 m. 1G over structured copper, 2 m within the 100 m limit for Cat6a patch lead.
Structured fibre
MPO-12 APC OM4 multi-mode trunk 46 44 x 800G Storage leaf to Storage spine at 30 m. 30 m run needs discrete optics over structured fiber. 800G OSFP VR8 (multi-mode) at Storage leaf and 800G OSFP VR8 (multi-mode) at Storage spine, over MPO-12 APC OM4 multi-mode trunk.
Racks
Rack 7 6 compute racks at 3 nodes each (fixed-limited) plus 1 network racks.
Power
Rack PDU 14 7 racks x 2 PDUs per rack.

Related