Fallen Off the Bus
A training job died overnight. Find the bad GPU, quarantine the node and get the run going again.
Briefingclick to collapse
Job 101 (a 7B run on 2 nodes) failed a few minutes ago and its owner is asking why. Your job:
- figure out which node and GPU killed it,
- drain that node so the scheduler stops placing work there,
- get the training job running again on healthy nodes.
Start with sacct, the job log (cat slurm-101.out) and the node's kernel log (ssh <node> then dmesg).
sim time 1:00energy 0.27 kWhcost $0.47carbon 0.10 kg CO₂e
Cluster4 × 8 H100 SXM 80GB · IB-NDR
node-1
idle
node-2
idle
node-3
idle
node-4
idle
0%100% util▲ throttled✕ lostclick a GPU for details
Mission: Fallen Off the Bus. You are ops@login-1 with sudo.
Type `help` for commands, `cat README.md` for notes. Objectives check themselves (right panel).
ops@login-1:~$
Objectives0/2 met
- node-2 is drained (with a reason)
- The 7B job is training again on 2 healthy nodes
−33 pts each
Alerts2 firing
GpuFellOffBus node-2/gpu6 · critical
Xid 79: GPU has fallen off the bus
JobFailed job 101 · warning
Job 101 (llama7b) failed with exit code 1:0
Dashboardslast 60s · hover for values
Training throughput0 tok/s
MFU (best job)0.0%
Mean GPU utilization0%
Facility power9.1 kW
Hottest GPU34°C