Seventy Billion Reasons

A 70B model on 32 GPUs: it doesn't fit. Choose a parallelism strategy that does, and runs fast.

Briefingclick to collapse

You have 4 nodes × 8 H100 (80 GB) and a 70B-parameter model to train. With plain data parallelism every GPU would need ~1.2 TB. It OOMs instantly.

Design a parallel layout that fits in memory and reaches ≥ 35% MFU on all 32 GPUs. Knobs in train.py: --tp (tensor parallel), --pp (pipeline parallel), --zero 0-3, --micro-batch, --grad-accum, --ckpt.

Remember: dp = 32 / (tp × pp), NVLink only exists inside a node, and 80 layers must split evenly across pipeline stages.

sim time 0:20energy 0.05 kWhcost $0.16carbon 0.02 kg CO₂e

Cluster4 × 8 H100 SXM 80GB · IB-NDR

node-1

idle

node-2

idle

node-3

idle

node-4

idle

0%100% util▲ throttled✕ lostclick a GPU for details

Mission: Seventy Billion Reasons. You are ops@login-1 with sudo.

Type `help` for commands, `cat README.md` for notes. Objectives check themselves (right panel).

ops@login-1:~$

Objectives0/2 met

A 70B job is training on 32 GPUs without OOM
…at ≥ 35% MFU

−33 pts each

Alerts1 firing

JobFailed job 101 · warning

Job 101 (llama70b) failed with exit code 1:0

Dashboardslast 20s · hover for values

Training throughput0 tok/s

MFU (best job)0.0%

Mean GPU utilization0%

Facility power9.2 kW

Hottest GPU33°C