Pipeline parallelism
Read it in Chapter 4 · checkpoint +50 XP →
Split the layer stack into stages on different GPUs and stream micro-batches through. Light traffic, but a bubble.
Pipeline parallelism (PP) assigns contiguous layers to stages. Stage 1 runs layers 1–20, stage 2 runs 21–40, and so on. Only the activations at stage boundaries cross the network, so PP is comfortable spanning nodes over InfiniBand & RoCE.
The bubble: at the start and end of each step some stages sit idle while the pipeline fills and drains. With p stages and m micro-batches per step (1F1B schedule):
bubble fraction ≈ (p − 1) / (m + p − 1)
More micro-batches (gradient accumulation) shrink the bubble. Interleaved schedules shrink it further, at the cost of more communication.
Constraints: the layer count must divide evenly (or nearly) into stages, and stages should be balanced. The first stage also holds the most in-flight activations.
The classic large-model recipe, 3D parallelism, combines all three: TP inside the node, PP across a few nodes, DP (often with ZeRO-1) across everything else.
Use it in a mission
Related
Sources & further reading
- GPipe (Huang et al., 2019)
- Megatron-LM (Shoeybi et al., 2019)
- Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
Written for LazyLab in our own words; sources are cited for depth and verification.