HBM & the memory hierarchy

Read it in Chapter 1 · checkpoint +50 XP →

High Bandwidth Memory: stacked DRAM next to the GPU die. Its capacity bounds model size and its bandwidth bounds many kernels.

High Bandwidth Memory stacks DRAM dies beside the GPU on a silicon interposer. An H100 SXM has 80 GB at ~3.35 TB/s. Below it sit L2 (tens of MB), per-SM shared memory/L1 and registers, each smaller and faster.

Capacity is the first wall in large-model training. Mixed-precision Adam needs ~16 bytes per parameter (weights, gradients, FP32 master copy and two moments), before activations. A 7B model needs ~107 GB of state alone, more than one GPU holds. That's why ZeRO/FSDP, Tensor parallelism and Activation checkpointing exist.

Bandwidth is the second wall. Any kernel with low arithmetic intensity (elementwise ops, norms, small matmuls) is limited by HBM bandwidth, not FLOPs. See Roofline model.

Try the calculator below: change the model, ZeRO stage and parallelism and watch the per-GPU footprint against the 80 GB line.

Use it in a mission

Related

Sources & further reading

• ZeRO (Rajbhandari et al., 2020)
• Reducing Activation Recomputation (Korthikanti et al., 2022)
• Modal GPU Glossary — hardware and software vocabulary
• NVIDIA H100 Tensor Core GPU

Written for LazyLab in our own words; sources are cited for depth and verification.