GB200 and GB300 NVL72 networking
Three networks, not one. Only two of them are a purchasing decision, and the GB200 and GB300 trays are not the same hardware.
The common misunderstanding about NVL72 is that NVLink replaces the cluster network. It does not. A rack carries three distinct networks and they do different jobs.
| Network | Scope | Per rack | Do you buy switches and optics? |
|---|---|---|---|
| NVLink | Inside the rack, GPU to GPU | 18 links per GPU to the in-rack NVSwitches, over a copper backplane, through 9 switch trays | No. Ships with the rack |
| ConnectX (east/west) | Between racks, GPU to GPU | 72 adapters, a 1:1 GPU-to-NIC ratio | Yes. This is the compute fabric |
| BlueField-3 (north/south) | Storage, provisioning, management | 18 or 36 links depending on generation | Yes. Storage and in-band fabrics |
NVLink is per-GPU and it is what makes 72 GPUs behave as one accelerator. It is also entirely internal: copper backplane, no optics, no cable schedule, no line in the bill of materials. You do not size it. You do size the other two.
GB200 and GB300 trays are different hardware
Both racks are 18 compute trays of 4 GPUs. The trays themselves differ in three ways that change the parts list.
| GB200 tray | GB300 tray | |
|---|---|---|
| East/west | 4× single-port 400G ConnectX-7 | 4× dual-port 800G ConnectX-8 |
| Rack network total | 28.8 Tb/s | 57.6 Tb/s |
| GPU to NIC path | GPU → Grace over C2C → NIC over PCIe | NIC-direct; the GPU reaches the ConnectX-8 itself |
| North/south | 2× dual-port BlueField-3, management ports bonded | 1× dual-port BlueField-3 B3240, no bond |
| North/south links per rack | 36 storage + 36 management | 18 storage + 18 management |
NVIDIA's own interface enumeration settles the DPU question. A GB200 tray presents M1 and M2 at PCI addresses enP6 and enP22, bonded as bond0 in LACP mode 4, plus S1 and S2 on storagenet. A GB300 tray presents only M1 and S1, both on enP22, the same address as the GB200's second card. The documentation states plainly that there is no bond configuration for GB300 trays, because there is only one card to bond.
A bond is redundancy, not consolidation. Two bonded management ports still land on two switch ports and still need two cables and two optics. Count a bond as one link and you halve the in-band fabric.
Ethernet changes the link count, not the bandwidth
ConnectX-8 runs one 800G port in InfiniBand mode and two 400G ports in Ethernet mode. A Spectrum-X build of a GB300 NVL72 therefore terminates 144 links at 400G instead of 72 at 800G. The rack still moves 57.6 Tb/s. It just presents twice as many terminations, and terminations are what switch, optic and cable counts are made of.
Rack granularity and power
The indivisible unit is a whole 72-GPU rack, so GPU counts round to multiples of 72 instead of 8. An NVL72 rack draws roughly 120 kW continuous at 132 kW nominal TDP, with peaks near 155 kW that the electrical design has to survive even though that is not the steady-state figure. It is liquid-cooled and needs a completely different power design from air-cooled DGX.