Data parallelism (DDP)

Read it in Chapter 4 · checkpoint +50 XP →

Every GPU holds a full model replica, processes different data and averages gradients each step.

The simplest way to scale: replicate the model on N GPUs, give each a different slice of the batch, then all-reduce the gradients so every replica takes the same optimizer step.

Cost: one all-reduce of the full gradient (2 bytes × params in BF16) per step. With ring all-reduce the per-GPU traffic is ~2× the gradient size, independent of N. That's why data parallelism scales so well.

Making it fast

• Overlap: DDP buckets gradients and all-reduces each bucket as soon as backward produces it, hiding communication behind compute (see CUDA streams & async execution).
• Gradient accumulation: run several micro-batches before syncing. You get a bigger global batch with the same memory and fewer syncs per token.

Limit: every GPU must hold the whole model state. Once that stops fitting, see ZeRO & FSDP, Tensor parallelism and Pipeline parallelism.

Use it in a mission

Related

Sources & further reading

• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
• NCCL User Guide

Written for LazyLab in our own words; sources are cited for depth and verification.