Night Shift
A GPU is busy with no job scheduled. Investigate, contain and remove persistence.
Briefingclick to collapse
Monitoring fired GpuBusyWithoutJob on node-2: a GPU is pegged, but Slurm says nothing is allocated there. This is a defensive incident-response drill on a simulated cluster.
Contain it properly:
- Stop the unauthorized workload.
- Close the way in. Something on node-2 is exposed to the network.
- Lock the compromised account.
- Remove persistence, so it doesn't come back by itself.
The objective only completes once the GPU stays clean for 2 minutes. Useful: ssh node-2, nvidia-smi, ps aux, ss -tnp, systemctl, journalctl -u <svc>, crontab -l -u <user>, sudo.
sim time 0:30energy 0.13 kWhcost $0.18carbon 0.05 kg CO₂e
Cluster4 × 8 H100 SXM 80GB · IB-NDR
node-1
job 101 · llama7b
node-2
idle
node-3
idle
node-4
idle
0%100% util▲ throttled✕ lostclick a GPU for details
Mission: Night Shift. You are ops@login-1 with sudo.
Type `help` for commands, `cat README.md` for notes. Objectives check themselves (right panel).
ops@login-1:~$
Objectives0/4 met
- Persistence removed (no malicious cron entry)
- Exposed Jupyter server stopped
- Account 'intern' locked
- No miner running for 2 minutes
−33 pts each
Alerts1 firing
GpuBusyWithoutJob node-2/gpu3 · warning
GPU 97% busy but no Slurm job is allocated here
Dashboardslast 30s · hover for values
Training throughput131.3k tok/s
MFU (best job)66.7%
Mean GPU utilization28%
Facility power16.4 kW
Hottest GPU77°C