Watching the Fleet

3.1nvidia-smi

nvidia-smi talks to the driver through NVML. The default table shows each GPU's temperature, power draw versus cap, memory use, utilization and the processes using it.

The views that answer real questions

nvidia-smi                                   # overview + processes
nvidia-smi -q -d PERFORMANCE                 # why are clocks low? (event reasons)
nvidia-smi -q -d TEMPERATURE,POWER,ECC       # thermals, power limits, memory errors
nvidia-smi --query-gpu=index,temperature.gpu,clocks.sm,power.draw --format=csv
nvidia-smi topo -m                           # NVLink/PCIe/NIC topology matrix
nvidia-smi nvlink -s                         # per-link NVLink status
sudo nvidia-smi -pl 500                      # set a power limit

Caveat: "GPU-Util" means a kernel was running during the sample window. It does not mean the GPU was busy. A GPU waiting at a barrier for a straggler still spins. For real saturation metrics use DCGM profiling fields such as SM activity and tensor pipe activity.

| N/A   85C    P0             437W / 700W |  70471MiB / 81560MiB |   98%      Default |

3.2Power & thermals

Every GPU has a power limit (TDP; 700 W for H100 SXM) and thermal thresholds. When it would exceed either, firmware lowers clocks:

• SW Power Cap: drawing the configured limit; clocks drop slightly. Often intentional. See Power capping.
• HW Slowdown / HW Thermal Slowdown: an emergency brake, typically from overheating (failed fan, blocked airflow, hot inlet air). Clocks can drop by half.

In synchronous training, one throttled GPU turns into a cluster-wide slowdown. Every collective waits for it. That's a straggler.

Where to look

• nvidia-smi -q -d PERFORMANCE shows "Clocks Event Reasons" (formerly throttle reasons).
• nvidia-smi -q -d TEMPERATURE shows current, slowdown and shutdown temperatures.
• DCGM fields such as DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE and the clock-throttle bitmask feed alerting.

    Clocks Event Reasons
        SW Power Cap                      : Not Active
        HW Slowdown                       : Active
            HW Thermal Slowdown           : Active

3.3Xid errors

An Xid is the driver's way of saying "this GPU had a problem". It appears in the kernel log (dmesg, journalctl -k) as NVRM: Xid (PCI:0000:xx:00.0): <code>, .... The messages are emitted by the kernel driver, which you can read in NVIDIA's open-gpu-kernel-modules.

Codes every GPU operator should recognize

Ops pattern: alert on Xid via DCGM, drain the node in Slurm with the Xid in the reason, requeue affected jobs from their checkpoints, then diagnose offline.

NVRM: Xid (PCI:0000:9a:00.0): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.

3.4ECC errors & row remapping

Data-center GPUs protect HBM (and on-chip SRAM) with ECC:

• Single-bit (correctable): fixed on the fly and counted. A slowly growing count is normal. A rapidly growing one is a warning.
• Double-bit (uncorrectable, DBE): the data is lost. The application gets an error (Xid 48 / 94 / 95) and the GPU usually needs a reset.

A100 and later GPUs use row remapping: a bad DRAM row is replaced with a spare at the next GPU reset. nvidia-smi -q -d ECC shows "Remapped Rows → Pending: Yes" until you reset. If remapping fails (spares exhausted), the GPU needs replacing.

Procedure: drain → wait for jobs to leave → nvidia-smi -r -i <gpu> → dcgmi diag -r 2 → return to service if clean.

3.5DCGM

DCGM runs a host engine (nv-hostengine) on each node and exposes:

• Telemetry fields: temperature, power, clocks, ECC, NVLink counters, Xid events and profiling metrics (SM_ACTIVE, SM_OCCUPANCY, PIPE_TENSOR_ACTIVE, DRAM_ACTIVE).
• Health watches that flag thermal, power, memory, PCIe and NVLink issues.
• Diagnostics: dcgmi diag -r 1|2|3, from quick deployment checks to stress tests. Run level 1 before a node rejoins the pool and level 3 after hardware work.
• dcgm-exporter for Prometheus, so it feeds Grafana dashboards and alert rules.

Alerts worth having on day one

• Any Xid event (critical ones page someone).
• HW thermal slowdown active for more than a minute.
• Uncorrectable ECC or pending row remap.
• GPU busy while the scheduler shows no job on that node. That catches leaks and Cryptojacking on GPU clusters.

3.6Stragglers

Every collective is a barrier. If one of 1,024 GPUs runs 40% slower (a thermal throttle, a flaky link, a noisy neighbor or a slow disk on one node), all 1,024 run 40% slower.

Signatures

• Throughput drops with no errors in the logs.
• Per-GPU utilization looks inverted: the straggler shows ~100%, the others less, because they idle at the barrier.
• The straggler's SM clock is low. nvidia-smi -q -d PERFORMANCE shows HW Thermal Slowdown or SW Power Cap.
• NCCL or framework timing shows one rank consistently arriving late to collectives.

Response: identify, drain, restart from a checkpoint on healthy hardware, and file a hardware ticket. At scale, automate it: continuous per-rank timing plus DCGM clocks and temperatures, and automatic cordoning.

index, utilization.gpu, clocks.sm
0, 61 %, 1980 MHz
1, 60 %, 1980 MHz
5, 99 %, 1089 MHz
6, 61 %, 1980 MHz