Anatomy of a GPU Cluster

1.1GPU

A CPU spends its transistors making one thread fast: deep caches, branch prediction, out-of-order execution. A GPU spends them on many lanes of arithmetic. It hides memory latency by keeping thousands of threads in flight and switching between them for free.

For ML and HPC that trade is ideal. The work is mostly dense linear algebra, the same operation over huge arrays. A data-center GPU such as the H100 pairs ~130 streaming multiprocessors with tensor cores and 80 GB of HBM at several TB/s.

What makes a GPU fast in practice

• Enough parallel work to fill every SM (see Occupancy).
• Arithmetic intensity high enough to be compute-bound, not memory-bound (see Roofline model).
• Data that stays on the device. PCIe transfers are ~50× slower than HBM.

In a cluster, the GPU is one node in a hierarchy: GPU → NVLink within a server → InfiniBand & RoCE between servers. Most cluster engineering is about keeping that hierarchy busy.

1.2Streaming Multiprocessor (SM)

An SM is roughly what a "core" is on a CPU, but built for throughput. Each SM has warp schedulers, a large register file, configurable shared memory / L1 and execution units: FP32/INT "CUDA cores", special-function units and tensor cores.

A kernel launch is split into thread blocks. Each block lands on one SM and stays there. The SM interleaves the warps of all resident blocks every cycle, so when one warp waits on memory another issues.

Why you care as an operator

• SM count × clock sets peak FLOP/s. A GPU throttled to 55% clock (thermal or power) delivers ~55% of the math. See Power & thermals.
• DCGM_FI_PROF_SM_ACTIVE and SM_OCCUPANCY (from DCGM) tell you whether SMs are actually busy. "GPU utilization" in nvidia-smi only means some kernel was running, not that every SM was.

1.3HBM & the memory hierarchy

High Bandwidth Memory stacks DRAM dies beside the GPU on a silicon interposer. An H100 SXM has 80 GB at ~3.35 TB/s. Below it sit L2 (tens of MB), per-SM shared memory/L1 and registers, each smaller and faster.

Capacity is the first wall in large-model training. Mixed-precision Adam needs ~16 bytes per parameter (weights, gradients, FP32 master copy and two moments), before activations. A 7B model needs ~107 GB of state alone, more than one GPU holds. That's why ZeRO/FSDP, Tensor parallelism and Activation checkpointing exist.

Bandwidth is the second wall. Any kernel with low arithmetic intensity (elementwise ops, norms, small matmuls) is limited by HBM bandwidth, not FLOPs. See Roofline model.

Try the calculator below: change the model, ZeRO stage and parallelism and watch the per-GPU footprint against the 80 GB line.

1.4PCIe

PCI Express connects GPUs to the host CPU, NICs and storage. A Gen5 x16 link moves ~64 GB/s per direction. Gen4 x16 moves ~32 GB/s. That's plenty for loading data, but slow for GPU-to-GPU traffic during training.

Why it matters

• GPUs without NVLink (e.g. L40S) must do all peer traffic over PCIe, often through the CPU's root complex. Tensor parallelism suffers badly.
• GPUDirect RDMA lets a NIC read and write GPU memory directly over PCIe, skipping host memory. Good servers pair each GPU with a NIC under the same PCIe switch. Check with nvidia-smi topo -m.
• "Fallen off the bus" (Xid 79) is literally the GPU disappearing from PCIe.

1.5NVLink

NVLink is a point-to-point interconnect between GPUs. An H100 SXM has 18 NVLink 4 links totalling ~900 GB/s bidirectional (~450 GB/s each way). Through NVSwitch, all 8 GPUs in an HGX server talk to each other at full speed.

Design rule: put the chattiest parallelism on the fastest link. Tensor parallelism does several all-reduces per layer, so it lives inside the NVLink domain. Data parallelism, which syncs once per step and overlaps with compute, can cross InfiniBand & RoCE.

Failure mode: NVLink errors appear as Xid 74. A degraded link forces traffic over PCIe and quietly slows collectives. nvidia-smi nvlink -s shows per-link status, and all_reduce_perf (nccl-tests) on an idle node shows the bandwidth drop.

$ nvidia-smi nvlink -s -i 2
GPU 2: NVIDIA H100 80GB HBM3
	 Link 3: 26.562 GB/s
	 Link 4: <inactive>
	 Link 5: 26.562 GB/s

1.6NVSwitch

Without a switch, NVLink is point-to-point, so GPUs only connect to a few neighbors. NVSwitch builds a non-blocking crossbar: in an HGX H100 server, four NVSwitch chips connect all 8 GPUs all-to-all at full NVLink bandwidth.

Newer systems extend the NVLink domain beyond a server (e.g. NVL72 racks), so 72 GPUs act like one large tensor-parallel island. The rule from NVLink still applies: the bigger the NVLink domain, the larger the tensor-parallel or expert-parallel groups you can run without touching the slower scale-out network.

nvidia-smi topo -m shows NV18 (18 bonded links) between every GPU pair in an NVSwitch system.

1.7InfiniBand & RoCE

Between servers, GPUs talk over a scale-out network. Two families dominate:

• InfiniBand (IB): a purpose-built RDMA fabric. NDR runs 400 Gb/s per port with very low latency and credit-based lossless flow control.
• RoCE v2: RDMA over Converged Ethernet. Same verbs API over Ethernet. It needs careful configuration (PFC/ECN) to stay lossless, but it's cheaper and familiar to network teams.

Modern training servers attach one 400G NIC per GPU (rail-optimized), so each GPU gets ~50 GB/s of scale-out bandwidth via GPUDirect RDMA.

Ops checklist

• ibstat should show every port Active / LinkUp at the expected rate.
• NCCL logs NET/IB when it uses RDMA and NET/Socket when it fell back to TCP, a classic silent 10× slowdown (often from NCCL_IB_DISABLE=1 or a bad HCA config).
• A port going down mid-job hangs collectives until the NCCL watchdog times out.

NCCL INFO NET/Socket : Using [0]eth0:10.0.1.12<0>
NCCL INFO NCCL_IB_DISABLE set by environment to 1.

1.8Cluster topology

A GPU cluster network is usually a fat tree. Leaf switches connect servers, and spine switches connect leaves. If uplink capacity equals downlink capacity the fabric is non-blocking (1:1). Cheaper designs oversubscribe (e.g. 2:1), which is fine for web traffic and painful for all-reduce, where everyone transmits at once.

Rail-optimized designs wire NIC i of every server to the same leaf switch ("rail i"). NCCL's ring and tree algorithms then keep most traffic on one rail, one hop away.

Rules of thumb

• The bandwidth hierarchy (HBM ≫ NVLink ≫ IB ≫ storage) should map onto the parallelism hierarchy. See Tensor parallelism, Pipeline parallelism and Data parallelism (DDP).
• Keep spare capacity. A warm spare node turns a hardware failure into a requeue instead of an outage. See Checkpointing & fault tolerance.
• Power and cooling are topology too. Racks have kW budgets, and liquid cooling changes what fits. See Energy & carbon.