Two Nodes, One Job

Your first multi-node training job won't even submit. Fix the batch script.

Briefingclick to collapse

A researcher left you train.sh: a 7B training run that should use 2 nodes × 8 H100s. They say "it just errors out." Get it running across both nodes, and make sure the nodes talk over InfiniBand, not the slow management Ethernet.

Useful commands: cat train.sh, nano train.sh, sinfo, sbatch train.sh, squeue, cat slurm-<jobid>.out.

sim time 0:00energy 0.00 kWhcost $0.00carbon 0.00 kg CO₂e

Cluster4 × 8 H100 SXM 80GB · IB-NDR

node-1

idle

node-2

idle

node-3

idle

node-4

idle

0%100% util▲ throttled✕ lostclick a GPU for details

Mission: Two Nodes, One Job. You are ops@login-1 with sudo.

Type `help` for commands, `cat README.md` for notes. Objectives check themselves (right panel).

ops@login-1:~$

Objectives0/2 met

A training job is running on 2 nodes × 8 GPUs
Inter-node traffic uses InfiniBand (≥ 200k tokens/s)

−33 pts each

Alertsall clear

No alerts. Quiet night.

Dashboardslast 0s · hover for values

Training throughput—

MFU (best job)—

Mean GPU utilization—

Facility power—

Hottest GPU—