OperationsMonitoring track
DCGM
Read it in Chapter 3 · checkpoint +50 XP →
Data Center GPU Manager: fleet-scale GPU telemetry, health checks, diagnostics and policy, the backbone of GPU monitoring.
DCGM runs a host engine (nv-hostengine) on each node and exposes:
- Telemetry fields: temperature, power, clocks, ECC, NVLink counters, Xid events and profiling metrics (
SM_ACTIVE,SM_OCCUPANCY,PIPE_TENSOR_ACTIVE,DRAM_ACTIVE). - Health watches that flag thermal, power, memory, PCIe and NVLink issues.
- Diagnostics:
dcgmi diag -r 1|2|3, from quick deployment checks to stress tests. Run level 1 before a node rejoins the pool and level 3 after hardware work. - dcgm-exporter for Prometheus, so it feeds Grafana dashboards and alert rules.
Alerts worth having on day one
- Any Xid event (critical ones page someone).
- HW thermal slowdown active for more than a minute.
- Uncorrectable ECC or pending row remap.
- GPU busy while the scheduler shows no job on that node. That catches leaks and Cryptojacking on GPU clusters.
Use it in a mission
Monitoring
The Slow Lane
Throughput just halved across 32 GPUs, but nothing failed. Find the straggler.
Security
Night Shift
A GPU is busy with no job scheduled. Investigate, contain and remove persistence.
Related
Sources & further reading
Written for LazyLab in our own words; sources are cited for depth and verification.