Scaling efficiency

Read it in Chapter 5 · checkpoint +50 XP →

Why 2× the GPUs rarely gives 2× the speed: serial work, communication and synchronization (Amdahl's law and friends).

Amdahl's law: if a fraction s of the work is serial, the speedup on n workers is at most 1 / (s + (1−s)/n). Even 5% serial work caps you at 20× speedup.

Distributed training adds communication that grows with scale: collective latency terms, cross-rack hops, and the growing chance that some GPU is slow (see Stragglers).

Weak vs strong scaling

• Strong scaling: same problem, more GPUs. Efficiency drops as per-GPU work shrinks.
• Weak scaling: grow the batch with the GPU count. Much friendlier, which is why large-batch training exists, within the limits of what still converges.

Measuring it: scaling efficiency = (throughput at n) / (n × throughput at 1). Report it alongside MFU. A job can scale perfectly and still be inefficient per GPU.

Related

Sources & further reading

• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
• PRACE Best Practice Guide – GPGPU — GPU programming and tuning practice for HPC

Written for LazyLab in our own words; sources are cited for depth and verification.