Checkpointing & fault tolerance

Read it in Chapter 2 · checkpoint +50 XP →

Periodically save model and optimizer state so a failure costs minutes of progress, not days.

At thousands of GPUs, something fails every few hours. Checkpointing bounds the damage: on failure, you restart from the last saved state and lose at most one checkpoint interval.

Choosing the interval: too frequent wastes I/O time, too rare wastes compute on failure. A classic approximation (Young/Daly) is interval ≈ √(2 × checkpoint_cost × MTBF).

Making it cheap

• Shard checkpoints: each rank writes its own slice in parallel.
• Async checkpointing: copy to host memory, then write in the background.
• Keep a few recent checkpoints. Corrupt or partial saves happen.

Ops integration: scontrol requeue or a job-level retry loop, plus a warm spare pool, turns "a GPU died" into a 5-minute blip. That's the loop you practice in Fallen Off the Bus and The Slow Lane.

Use it in a mission

Related

Sources & further reading

• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory
• Slurm documentation

Written for LazyLab in our own words; sources are cited for depth and verification.