LazyLabES
InterconnectArchitecture track

NVLink

Read it in Chapter 1 · checkpoint +50 XP →

NVIDIA's high-bandwidth GPU-to-GPU link: ~450 GB/s per direction per H100, roughly 7× PCIe Gen5.

NVLink is a point-to-point interconnect between GPUs. An H100 SXM has 18 NVLink 4 links totalling ~900 GB/s bidirectional (~450 GB/s each way). Through NVSwitch, all 8 GPUs in an HGX server talk to each other at full speed.

Design rule: put the chattiest parallelism on the fastest link. Tensor parallelism does several all-reduces per layer, so it lives inside the NVLink domain. Data parallelism, which syncs once per step and overlaps with compute, can cross InfiniBand & RoCE.

Failure mode: NVLink errors appear as Xid 74. A degraded link forces traffic over PCIe and quietly slows collectives. nvidia-smi nvlink -s shows per-link status, and all_reduce_perf (nccl-tests) on an idle node shows the bandwidth drop.

LIVE All-reduce cost model
ringtree
1 µs100 µs10.0 ms1.0 s1 KB100 KB10 MB1 GBmessage sizetimetreeringyou
Ring
14.0 ms
Tree
13.4 ms
Algorithm bw
23.6 GB/s
Bus bw (ring)
43.7 GB/s

α-β model: ring = 2(n−1)/n·S/B + 2(n−1)·α; tree ≈ 2·S/B + 2·log₂(n)·α. Real NCCL picks per message size and topology.

Use it in a mission

Optimization
Seventy Billion Reasons
A 70B model on 32 GPUs: it doesn't fit. Choose a parallelism strategy that does, and runs fast.
Architecture
Build the Pod
Spec a GPU pod that trains a 13B model in two weeks, within budget and the facility's power envelope.

Related

NVSwitchTensor parallelismAll-reduce (ring & tree)PCIe

Sources & further reading

Written for LazyLab in our own words; sources are cited for depth and verification.