Feed the Beast
16 H100s running at 26% MFU. Tune the training config past 55% without running out of memory.
Briefingclick to collapse
Job 101 trains a 7B model on 2 nodes × 8 H100s, but Model FLOPs Utilization is about 26%. The config in train.sh was copied from a tiny debugging run.
Get the job to ≥ 55% MFU. Every flag in the train.py line is fair game: --micro-batch, --grad-accum, --overlap/--no-overlap, --ckpt/--no-ckpt, --workers, --precision, --zero, --tp.
Workflow: nano train.sh → save → scontrol requeue 101 (re-reads the script) → watch tail slurm-101.out. Look for OOM errors and the "dataloader wait" marker.
sim time 0:40energy 0.12 kWhcost $0.46carbon 0.05 kg CO₂e
Cluster2 × 8 H100 SXM 80GB · IB-NDR
node-1
job 101 · llama7b
node-2
job 101 · llama7b
0%100% util▲ throttled✕ lostclick a GPU for details
Mission: Feed the Beast. You are ops@login-1 with sudo.
Type `help` for commands, `cat README.md` for notes. Objectives check themselves (right panel).
ops@login-1:~$
Objectives0/2 met
- The 7B job is training at ≥ 55% MFU
- …still using both nodes (16 GPUs)
−33 pts each
Alertsall clear
No alerts. Quiet night.
Dashboardslast 40s · hover for values
Training throughput102.0k tok/s
MFU (best job)25.9%
Mean GPU utilization59%
Facility power12.3 kW
Hottest GPU60°C