ParallelismOptimization track
Mixed precision (BF16, FP8)
Read it in Chapter 4 · checkpoint +50 XP →
Do the math in 16- or 8-bit formats on tensor cores and keep an FP32 master copy for stable updates.
Mixed-precision training runs forward and backward in a low-precision format and keeps FP32 master weights for the optimizer update:
- BF16 keeps FP32's exponent range, so it rarely needs loss scaling. It's the default on A100/H100.
- FP16 needs dynamic loss scaling to keep small gradients from underflowing.
- FP8 (H100+) roughly doubles matmul throughput again. Libraries like Transformer Engine manage per-tensor scaling.
Why it's a big deal: Tensor cores deliver ~15× more FLOP/s in BF16 than plain FP32 on H100. Training in FP32 leaves almost all of the GPU idle.
Memory: weights and gradients in 2 bytes, plus 12 bytes of FP32 optimizer state (master, Adam m, v) gives the "16 bytes per parameter" rule used throughout the Atlas.
Use it in a mission
Related
Sources & further reading
Written for LazyLab in our own words; sources are cited for depth and verification.