The Slow Lane
Throughput just halved across 32 GPUs, but nothing failed. Find the straggler.
Briefingclick to collapse
A 7B run is training on 4 nodes (32 H100s), and node-5 is idle as a spare. A few minutes ago tokens/s dropped by ~45%. No errors in the job log. Nothing crashed.
In synchronous data-parallel training, every step waits for the slowest GPU. Find it, take its node out of rotation and get MFU back above 60%.
Tools: the dashboards on the right, nvidia-smi, nvidia-smi -q -d PERFORMANCE, --query-gpu=index,temperature.gpu,clocks.sm, scontrol.
sim time 1:30energy 0.76 kWhcost $2.09carbon 0.29 kg CO₂e
Cluster5 × 8 H100 SXM 80GB · IB-NDR
node-1
job 101 · llama7b
node-2
job 101 · llama7b
node-3
job 101 · llama7b
node-4
job 101 · llama7b
node-5
idle
0%100% util▲ throttled✕ lostclick a GPU for details
Mission: The Slow Lane. You are ops@login-1 with sudo.
Type `help` for commands, `cat README.md` for notes. Objectives check themselves (right panel).
ops@login-1:~$
Objectives0/2 met
- The node with the throttling GPU is drained
- The training job is back above 60% MFU on healthy nodes
−33 pts each
Alerts1 firing
GpuThermalSlowdown node-3/gpu5 · warning
HW thermal slowdown active at 85°C
Dashboardslast 90s · hover for values
Training throughput283.9k tok/s
MFU (best job)36.1%
Mean GPU utilization44%
Facility power25.5 kW
Hottest GPU85°C