InterconnectArchitecture track
Cluster topology
Read it in Chapter 1 · checkpoint +50 XP →
How nodes, switches and links are wired: fat trees, rail-optimized designs, and why oversubscription bites collectives.
A GPU cluster network is usually a fat tree. Leaf switches connect servers, and spine switches connect leaves. If uplink capacity equals downlink capacity the fabric is non-blocking (1:1). Cheaper designs oversubscribe (e.g. 2:1), which is fine for web traffic and painful for all-reduce, where everyone transmits at once.
Rail-optimized designs wire NIC i of every server to the same leaf switch ("rail i"). NCCL's ring and tree algorithms then keep most traffic on one rail, one hop away.
Rules of thumb
- The bandwidth hierarchy (HBM ≫ NVLink ≫ IB ≫ storage) should map onto the parallelism hierarchy. See Tensor parallelism, Pipeline parallelism and Data parallelism (DDP).
- Keep spare capacity. A warm spare node turns a hardware failure into a requeue instead of an outage. See Checkpointing & fault tolerance.
- Power and cooling are topology too. Racks have kW budgets, and liquid cooling changes what fits. See Energy & carbon.
Use it in a mission
Related
Sources & further reading
- Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
- NCCL User Guide
Written for LazyLab in our own words; sources are cited for depth and verification.