InfiniBand against Spectrum-X Ethernet
Both are production choices with published reference architectures. What differs, concretely, in the parts list.
This page is not an argument for one or the other. NVIDIA publishes reference architectures for both, Cisco publishes validated designs for lossless Ethernet AI fabrics, and both are deployed at scale. The useful question is what actually changes when you switch, because it is more than the switch model.
The port arrangement changes
ConnectX-8 runs one 800G port in InfiniBand mode and two 400G ports in Ethernet mode. Same adapter, same bandwidth, different number of links.
An Ethernet build of a given set of servers therefore terminates roughly twice as many links at half the speed. Aggregate bandwidth is unchanged. Switch port count, optic count and cable count are not. This is the largest single difference between the two bills of materials and it has nothing to do with the protocol.
Ethernet designs commonly use multiple planes
The DGX B300 Spectrum-X reference uses two independent planes, with 2×400GbE per GPU reaching one plane each. That is a different topology from a single-plane fabric with twice the rails: rails are divided between planes, and each plane is sized as its own fabric. It is also what lets a 512-node SuperPOD stay in two tiers.
Congestion control is not optional on Ethernet
InfiniBand has credit-based flow control in the fabric. Ethernet has to be made lossless, which in practice means RoCEv2 with PFC and ECN configured correctly, and increasingly adaptive routing in place of static ECMP hashing. ECMP over a handful of very large elephant flows collides, and one collision on an AI collective stalls every rank in the job.
Adaptive routing is not NVIDIA-only. Cisco Silicon One does per-packet load balancing (PLB) and flowlet load balancing (DLB), and an add-on licence brings NVIDIA Spectrum-X adaptive routing to Nexus 9300 with NVIDIA SuperNICs. Both are validated combinations.
Silicon does not follow the badge
Vendor and silicon are separate questions. The Cisco Nexus 9100 series is built on NVIDIA Spectrum-4 silicon, so choosing Cisco for the chassis, software and support model does not necessarily mean choosing away from Spectrum-X. It is not the binary decision it looks like.
Practical summary
| InfiniBand | Spectrum-X Ethernet | |
|---|---|---|
| ConnectX-8 port mode | 1×800G | 2×400G |
| Links per adapter | 1 | 2 |
| Typical plane count | 1 | 2 in the B300 reference |
| Losslessness | Credit-based, in-fabric | RoCEv2 with PFC and ECN, configured |
| Load balancing | Adaptive routing | Adaptive routing / PLB / DLB, licensed on some platforms |