GPU

Read it in Chapter 1 · checkpoint +50 XP →

A throughput processor: thousands of simple lanes that hide latency with parallelism instead of caches and prediction.

A CPU spends its transistors making one thread fast: deep caches, branch prediction, out-of-order execution. A GPU spends them on many lanes of arithmetic. It hides memory latency by keeping thousands of threads in flight and switching between them for free.

For ML and HPC that trade is ideal. The work is mostly dense linear algebra, the same operation over huge arrays. A data-center GPU such as the H100 pairs ~130 streaming multiprocessors with tensor cores and 80 GB of HBM at several TB/s.

What makes a GPU fast in practice

• Enough parallel work to fill every SM (see Occupancy).
• Arithmetic intensity high enough to be compute-bound, not memory-bound (see Roofline model).
• Data that stays on the device. PCIe transfers are ~50× slower than HBM.

In a cluster, the GPU is one node in a hierarchy: GPU → NVLink within a server → InfiniBand & RoCE between servers. Most cluster engineering is about keeping that hierarchy busy.

Use it in a mission

Related

Sources & further reading

• Modal GPU Glossary — hardware and software vocabulary
• PRACE Best Practice Guide – GPGPU — GPU programming and tuning practice for HPC
• CUDA C++ Programming Guide

Written for LazyLab in our own words; sources are cited for depth and verification.