PCIe

Read it in Chapter 1 · checkpoint +50 XP →

The general-purpose bus between CPU, GPU, NIC and NVMe. Ubiquitous, but an order of magnitude slower than NVLink.

PCI Express connects GPUs to the host CPU, NICs and storage. A Gen5 x16 link moves ~64 GB/s per direction. Gen4 x16 moves ~32 GB/s. That's plenty for loading data, but slow for GPU-to-GPU traffic during training.

Why it matters

• GPUs without NVLink (e.g. L40S) must do all peer traffic over PCIe, often through the CPU's root complex. Tensor parallelism suffers badly.
• GPUDirect RDMA lets a NIC read and write GPU memory directly over PCIe, skipping host memory. Good servers pair each GPU with a NIC under the same PCIe switch. Check with nvidia-smi topo -m.
• "Fallen off the bus" (Xid 79) is literally the GPU disappearing from PCIe.

Related

Sources & further reading

• Modal GPU Glossary — hardware and software vocabulary
• PRACE Best Practice Guide – GPGPU — GPU programming and tuning practice for HPC

Written for LazyLab in our own words; sources are cited for depth and verification.