HardwareMonitoring track
Power & thermals
Read it in Chapter 3 · checkpoint +50 XP →
GPUs throttle clocks to stay under their power limit and temperature thresholds, and the slowest GPU sets the pace.
Every GPU has a power limit (TDP; 700 W for H100 SXM) and thermal thresholds. When it would exceed either, firmware lowers clocks:
- SW Power Cap: drawing the configured limit; clocks drop slightly. Often intentional. See Power capping.
- HW Slowdown / HW Thermal Slowdown: an emergency brake, typically from overheating (failed fan, blocked airflow, hot inlet air). Clocks can drop by half.
In synchronous training, one throttled GPU turns into a cluster-wide slowdown. Every collective waits for it. That's a straggler.
Where to look
nvidia-smi -q -d PERFORMANCEshows "Clocks Event Reasons" (formerly throttle reasons).nvidia-smi -q -d TEMPERATUREshows current, slowdown and shutdown temperatures.- DCGM fields such as
DCGM_FI_DEV_GPU_TEMP,DCGM_FI_DEV_POWER_USAGEand the clock-throttle bitmask feed alerting.
Use it in a mission
Monitoring
The Slow Lane
Throughput just halved across 32 GPUs, but nothing failed. Find the straggler.
Performance
Watts Up
The facility asks for a 20% power cut. Keep at least 85% of the throughput.
Related
Sources & further reading
Written for LazyLab in our own words; sources are cited for depth and verification.