NCCL

Read it in Chapter 2 · checkpoint +50 XP →

NVIDIA's collective communication library: topology-aware all-reduce, all-gather and friends over NVLink, PCIe and RDMA.

NCCL ("nickel") implements the collectives distributed training runs on: all-reduce, all-gather, reduce-scatter, broadcast and send/recv. It discovers the topology (NVLink, NVSwitch, PCIe, NICs) and builds rings and trees to use it. See All-reduce (ring & tree).

Environment variables worth knowing

• NCCL_DEBUG=INFO prints the chosen transport and channels. Read it.
• NCCL_IB_DISABLE=1 forces TCP sockets instead of InfiniBand. It's useful on laptops and a disaster in production.
• NCCL_SOCKET_IFNAME / NCCL_IB_HCA select interfaces.

Diagnosing

• all_reduce_perf from nccl-tests measures achievable bus bandwidth. Run it on idle nodes to catch bad links before users do.
• Hung collectives end with a PyTorch watchdog timeout. The cause is usually one bad rank: a dead link, a lost GPU or a straggler that stopped responding.

Use it in a mission

Related

Sources & further reading

• NCCL User Guide
• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory

Written for LazyLab in our own words; sources are cited for depth and verification.