InterconnectArchitecture track
InfiniBand & RoCE
Read it in Chapter 1 · checkpoint +50 XP →
The scale-out fabric: RDMA networks that move tensors between servers without involving the CPU.
Between servers, GPUs talk over a scale-out network. Two families dominate:
- InfiniBand (IB): a purpose-built RDMA fabric. NDR runs 400 Gb/s per port with very low latency and credit-based lossless flow control.
- RoCE v2: RDMA over Converged Ethernet. Same verbs API over Ethernet. It needs careful configuration (PFC/ECN) to stay lossless, but it's cheaper and familiar to network teams.
Modern training servers attach one 400G NIC per GPU (rail-optimized), so each GPU gets ~50 GB/s of scale-out bandwidth via GPUDirect RDMA.
Ops checklist
ibstatshould show every portActive/LinkUpat the expected rate.- NCCL logs
NET/IBwhen it uses RDMA andNET/Socketwhen it fell back to TCP, a classic silent 10× slowdown (often fromNCCL_IB_DISABLE=1or a bad HCA config). - A port going down mid-job hangs collectives until the NCCL watchdog times out.
Use it in a mission
Running
Two Nodes, One Job
Your first multi-node training job won't even submit. Fix the batch script.
Architecture
Build the Pod
Spec a GPU pod that trains a 13B model in two weeks, within budget and the facility's power envelope.
Related
Sources & further reading
- NCCL User Guide
- Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
- Modal GPU Glossary — hardware and software vocabulary
Written for LazyLab in our own words; sources are cited for depth and verification.