Measuring and Scaling

5.1Roofline model

Every kernel has an arithmetic intensity (AI): FLOPs performed per byte moved from memory. The roofline says:

attainable FLOP/s = min( peak FLOP/s ,  AI × memory bandwidth )

Plot it on log-log axes and you get a slanted "memory roof" meeting a flat "compute roof" at the ridge point. For an H100 in BF16 that's ~989 TFLOP/s ÷ 3.35 TB/s ≈ 295 FLOPs/byte.

• Elementwise ops (add, GELU, layer norm) have AI around 1. They're hopelessly memory-bound, so fuse them.
• Large matmuls have AI in the thousands. They're compute-bound, and that's where tensor cores shine.
• Attention without fusion is memory-bound. FlashAttention restructures it to stay on-chip.

The PRACE GPGPU guide uses the same reasoning for HPC kernels: first decide which roof you're under, then optimize for that resource.

5.2MFU (Model FLOPs Utilization)

For dense transformers, training costs about 6 × parameters × tokens FLOPs (2 forward + 4 backward). MFU compares what you achieved to the theoretical peak:

MFU = (6 · N · tokens_per_second) / (num_GPUs · peak_FLOP/s)

It deliberately ignores recomputation from Activation checkpointing (that's HFU, hardware FLOPs utilization), so it only rewards useful work.

What's good? 35–55% is typical for well-tuned large-model training on H100-class clusters. Above 60% is excellent.

What drags it down: small micro-batches, exposed communication (no overlap), pipeline bubbles, data-loader stalls, Stragglers, throttled clocks and plain FP32. The Feed the Beast mission walks through these in order.

5.3All-reduce (ring & tree)

Ring all-reduce arranges n ranks in a ring and runs two phases (reduce-scatter, then all-gather) of n−1 steps each. Every rank sends and receives ~2(n−1)/n × the buffer size:

T_ring ≈ 2(n−1)/n · S/B  +  2(n−1) · α

where S is bytes, B is per-link bandwidth and α is per-step latency. The bandwidth term barely grows with n. The latency term does, which hurts small messages at large scale.

Tree algorithms (NCCL's double binary tree) cut the latency term to ~log₂(n) steps and win for small messages and very large rank counts.

Bus bandwidth, the number nccl-tests reports, normalizes away the 2(n−1)/n factor, so you can compare it straight to link speed. A healthy NVLink node shows hundreds of GB/s. A node with a degraded link shows PCIe-class numbers.

Use the widget to see how size, rank count and bandwidth trade off.

5.4Scaling efficiency

Amdahl's law: if a fraction s of the work is serial, the speedup on n workers is at most 1 / (s + (1−s)/n). Even 5% serial work caps you at 20× speedup.

Distributed training adds communication that grows with scale: collective latency terms, cross-rack hops, and the growing chance that some GPU is slow (see Stragglers).

Weak vs strong scaling

• Strong scaling: same problem, more GPUs. Efficiency drops as per-GPU work shrinks.
• Weak scaling: grow the batch with the GPU count. Much friendlier, which is why large-batch training exists, within the limits of what still converges.

Measuring it: scaling efficiency = (throughput at n) / (n × throughput at 1). Report it alongside MFU. A job can scale perfectly and still be inefficient per GPU.

5.5Input pipeline

Once the model math is fast, the CPU side often becomes the limit: reading shards from storage, decompressing, tokenizing, shuffling and collating batches.

Symptoms: GPU utilization saw-tooths, step time doesn't improve when you optimize the model, and logs show time waiting on the next batch.

Fixes

• More data-loader workers (processes) per rank, and pinned memory.
• Prefetching: prepare batch k+1 while batch k trains (see CUDA streams & async execution).
• Pre-tokenize offline and store packed binary shards. It's the single biggest win for LLM pretraining.
• Make sure the parallel filesystem or object store can deliver the aggregate bandwidth: GPUs × tokens/s × bytes per token, plus checkpoint traffic.

The Feed the Beast mission shows the bottleneck moving to data loading the moment communication is fixed.

[rank0] step 120 | loss 3.112 | 1.02 s/step | 128.4k tok/s | MFU 32.5% | dataloader wait

5.6Power capping

Near the top of the voltage/frequency curve, power grows much faster than performance: dynamic power ∝ f·V², and voltage must rise with frequency. Capping power therefore costs proportionally less performance than it saves.

In training, only the compute-bound part of each step slows down. Communication, data loading and pipeline bubbles don't care about SM clocks, so the real-world throughput hit is smaller still.

How

sudo nvidia-smi -pl 550            # all GPUs on this node
sudo nvidia-smi -pl 550 -i 3       # just GPU 3

Caps reset on reboot unless managed by your provisioning or by DCGM policies.

Caveats: cap all GPUs in a job. Capping one creates a straggler. And measure tokens per joule, not just tokens per second.

5.7Energy & carbon

A GPU cluster is a power plant's customer. A rough accounting:

facility energy (kWh) = IT energy (GPUs + hosts + network) × PUE
emissions (kg CO₂e)   = facility energy × grid carbon intensity

• PUE (Power Usage Effectiveness) captures cooling and power-delivery overhead: ~1.1 for the best liquid-cooled sites, 1.3–1.6 for typical ones.
• Carbon intensity varies by more than 10× between regions and by time of day. Where and when you train can matter as much as how.

Levers you control

• Efficiency first: higher MFU means fewer GPU-hours for the same model.
• Power caps trade a little speed for a lot of energy.
• Right-size: don't leave idle allocations running. Schedulers can reclaim them.
• Report kWh and CO₂e per run, next to cost and accuracy, so trade-offs are visible.

Every LazyLab run shows its energy, cost and carbon in the header.