Stragglers

Read it in Chapter 3 · checkpoint +50 XP →

In synchronous training the whole job runs at the speed of its slowest GPU. Finding that GPU is a core ops skill.

Every collective is a barrier. If one of 1,024 GPUs runs 40% slower (a thermal throttle, a flaky link, a noisy neighbor or a slow disk on one node), all 1,024 run 40% slower.

Signatures

• Throughput drops with no errors in the logs.
• Per-GPU utilization looks inverted: the straggler shows ~100%, the others less, because they idle at the barrier.
• The straggler's SM clock is low. nvidia-smi -q -d PERFORMANCE shows HW Thermal Slowdown or SW Power Cap.
• NCCL or framework timing shows one rank consistently arriving late to collectives.

Response: identify, drain, restart from a checkpoint on healthy hardware, and file a hardware ticket. At scale, automate it: continuous per-rank timing plus DCGM clocks and temperatures, and automatic cordoning.

Use it in a mission

Related

Sources & further reading

• NVIDIA DCGM documentation
• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory

Written for LazyLab in our own words; sources are cited for depth and verification.