All-reduce (ring & tree)

Read it in Chapter 5 · checkpoint +50 XP →

Sum a buffer across all ranks and give everyone the result. The collective that data-parallel training lives on.

Ring all-reduce arranges n ranks in a ring and runs two phases (reduce-scatter, then all-gather) of n−1 steps each. Every rank sends and receives ~2(n−1)/n × the buffer size:

T_ring ≈ 2(n−1)/n · S/B  +  2(n−1) · α

where S is bytes, B is per-link bandwidth and α is per-step latency. The bandwidth term barely grows with n. The latency term does, which hurts small messages at large scale.

Tree algorithms (NCCL's double binary tree) cut the latency term to ~log₂(n) steps and win for small messages and very large rank counts.

Bus bandwidth, the number nccl-tests reports, normalizes away the 2(n−1)/n factor, so you can compare it straight to link speed. A healthy NVLink node shows hundreds of GB/s. A node with a degraded link shows PCIe-class numbers.

Use the widget to see how size, rank count and bandwidth trade off.

Use it in a mission

Related

Sources & further reading

• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
• NCCL User Guide

Written for LazyLab in our own words; sources are cited for depth and verification.