nvidia-smi

Read it in Chapter 3 · checkpoint +50 XP →

The first tool you reach for: per-GPU utilization, memory, temperature, power, clocks, processes and configuration.

nvidia-smi talks to the driver through NVML. The default table shows each GPU's temperature, power draw versus cap, memory use, utilization and the processes using it.

The views that answer real questions

nvidia-smi                                   # overview + processes
nvidia-smi -q -d PERFORMANCE                 # why are clocks low? (event reasons)
nvidia-smi -q -d TEMPERATURE,POWER,ECC       # thermals, power limits, memory errors
nvidia-smi --query-gpu=index,temperature.gpu,clocks.sm,power.draw --format=csv
nvidia-smi topo -m                           # NVLink/PCIe/NIC topology matrix
nvidia-smi nvlink -s                         # per-link NVLink status
sudo nvidia-smi -pl 500                      # set a power limit

Caveat: "GPU-Util" means a kernel was running during the sample window. It does not mean the GPU was busy. A GPU waiting at a barrier for a straggler still spins. For real saturation metrics use DCGM profiling fields such as SM activity and tensor pipe activity.

Use it in a mission

Related

Sources & further reading

• nvidia-smi documentation

Written for LazyLab in our own words; sources are cited for depth and verification.