OperationsMonitoring track
Xid errors
Read it in Chapter 3 · checkpoint +50 XP →
Error codes the NVIDIA kernel driver writes to the system log when something goes wrong on a GPU.
An Xid is the driver's way of saying "this GPU had a problem". It appears in the kernel log (dmesg, journalctl -k) as NVRM: Xid (PCI:0000:xx:00.0): <code>, .... The messages are emitted by the kernel driver, which you can read in NVIDIA's open-gpu-kernel-modules.
Codes every GPU operator should recognize
| Xid | Meaning | Typical action |
|---|---|---|
| 13, 31, 43 | Application errors (bad memory access, illegal instruction) | Usually the user's code |
| 48 | Uncorrectable (double-bit) ECC error | Reset GPU, check row remapping, watch for repeats |
| 63, 64 | Row-remapping events | Reset when idle; RMA if remapping fails |
| 74 | NVLink error | Check links, reseat, run nccl-tests |
| 79 | GPU has fallen off the bus | Drain node, reboot, likely hardware ticket |
| 94, 95 | Contained / uncontained ECC error | Contained: affected app only. Uncontained: reset |
Ops pattern: alert on Xid via DCGM, drain the node in Slurm with the Xid in the reason, requeue affected jobs from their checkpoints, then diagnose offline.
Use it in a mission
Related
Sources & further reading
- NVIDIA Xid Errors
- NVIDIA open-gpu-kernel-modules — open-source Linux GPU kernel driver
- NVIDIA DCGM documentation
Written for LazyLab in our own words; sources are cited for depth and verification.