AI Network Designer Design a network →

Fat-tree sizing and tier boundaries

When two tiers run out, why the answer is much larger than P²/2 suggests, and what breakout does to it.

A non-blocking fat tree built from switches with P ports supports:

Half the ports on each leaf face down and half face up. That is the division by two. For a 64-port switch it works out at 2,048 endpoints in two tiers and 65,536 in three.

The boundary is per rail, not per cluster

This is where people trip. In a rail-optimized fabric each rail is its own fat tree, and each rail carries one endpoint per node, not one per GPU. The two-tier ceiling of 2,048 endpoints is therefore 2,048 nodes.

At eight GPUs per node that is 16,384 GPUs still inside a two-tier fabric on 64-port switches. Read the cap as a GPU count and you under-estimate the reachable scale by a factor of the GPUs per node, then add a third tier years before you need one.

Breakout moves the boundary again

A switch port faster than the endpoint it serves can be broken out if the speeds divide cleanly: an 800G port becomes 2×400G, 4×200G or 8×100G. A 64-port 800G switch serving 400G NICs is effectively a 128-port switch. P²/2 is quadratic, so the two-tier ceiling jumps from 2,048 nodes to 8,192.

Breakout has limits. The switch port must be at least as fast as the endpoint, the division must be exact, and the practical ceiling is around 8:1. A design that assumes arbitrary breakout produces port counts nobody can cable.

Cages against logical ports

Port count and optic count are different numbers. A QM9700 presents 64 logical 400G ports through only 32 physical OSFP cages, because each cage carries a twin-port 800G module terminating two 400G links.

Order one transceiver per logical port and you buy twice what you need. Order one per cage without accounting for the split and the link count is wrong. The cage count also bounds how much breakout a switch can present, since breakout multiplies out of cages, not logical ports.

What a third tier costs

A core tier is not simply more switches. Spine switches now split their own ports between downlinks to leaves and uplinks to core, so a spine in a three-tier fabric serves roughly half the leaves it served in a two-tier one. Size the spine tier with its full port count available for downlinks and you overcommit every spine switch. The fabric cannot then be cabled as drawn.

Related