Power capping
Read it in Chapter 5 · checkpoint +50 XP →
Lower a GPU's power limit and give up a little speed for a lot less energy: often 20% less power for under 10% slower.
Near the top of the voltage/frequency curve, power grows much faster than performance: dynamic power ∝ f·V², and voltage must rise with frequency. Capping power therefore costs proportionally less performance than it saves.
In training, only the compute-bound part of each step slows down. Communication, data loading and pipeline bubbles don't care about SM clocks, so the real-world throughput hit is smaller still.
How
sudo nvidia-smi -pl 550 # all GPUs on this node sudo nvidia-smi -pl 550 -i 3 # just GPU 3
Caps reset on reboot unless managed by your provisioning or by DCGM policies.
Caveats: cap all GPUs in a job. Capping one creates a straggler. And measure tokens per joule, not just tokens per second.
Use it in a mission
Related
Sources & further reading
Written for LazyLab in our own words; sources are cited for depth and verification.