OperationsMonitoring track
ECC errors & row remapping
Read it in Chapter 3 · checkpoint +50 XP →
HBM is protected by error-correcting codes: single-bit errors are fixed silently, double-bit errors stop the work.
Data-center GPUs protect HBM (and on-chip SRAM) with ECC:
- Single-bit (correctable): fixed on the fly and counted. A slowly growing count is normal. A rapidly growing one is a warning.
- Double-bit (uncorrectable, DBE): the data is lost. The application gets an error (Xid 48 / 94 / 95) and the GPU usually needs a reset.
A100 and later GPUs use row remapping: a bad DRAM row is replaced with a spare at the next GPU reset. nvidia-smi -q -d ECC shows "Remapped Rows → Pending: Yes" until you reset. If remapping fails (spares exhausted), the GPU needs replacing.
Procedure: drain → wait for jobs to leave → nvidia-smi -r -i <gpu> → dcgmi diag -r 2 → return to service if clean.
Related
Sources & further reading
Written for LazyLab in our own words; sources are cited for depth and verification.