Glossary

A throughput processor: thousands of simple lanes that hide latency with parallelism instead of caches and prediction.

The GPU's core building block: schedulers, registers, shared memory, CUDA cores and tensor cores.

32 threads that execute in lockstep. Divergent branches and scattered memory access cost real performance.

Matrix-multiply-accumulate units that deliver most of a modern GPU's FLOPs, if you feed them the right precision and shapes.

High Bandwidth Memory: stacked DRAM next to the GPU die. Its capacity bounds model size and its bandwidth bounds many kernels.

Hardware partitioning of one GPU into up to seven isolated instances, each with its own SMs, L2 slices and memory.

GPUs throttle clocks to stay under their power limit and temperature thresholds, and the slowest GPU sets the pace.

The general-purpose bus between CPU, GPU, NIC and NVMe. Ubiquitous, but an order of magnitude slower than NVLink.

NVIDIA's high-bandwidth GPU-to-GPU link: ~450 GB/s per direction per H100, roughly 7× PCIe Gen5.

A switch chip that gives every GPU in a server full-bandwidth NVLink to every other GPU.

The scale-out fabric: RDMA networks that move tensors between servers without involving the CPU.

How nodes, switches and links are wired: fat trees, rail-optimized designs, and why oversubscription bites collectives.

Kernel driver → user-mode driver → CUDA runtime → libraries → frameworks. Version mismatches live at every seam.

Queues of GPU work that let copies, kernels and communication overlap instead of waiting on each other.

NVIDIA's collective communication library: topology-aware all-reduce, all-gather and friends over NVLink, PCIe and RDMA.

The batch scheduler that runs most HPC and many AI clusters: partitions, jobs, allocations and node states.

Running GPU workloads on Kubernetes: device plugins, the GPU Operator, MIG profiles and gang scheduling.

How containers get GPU access: the host driver, injected device nodes and the NVIDIA Container Toolkit.

Every GPU holds a full model replica, processes different data and averages gradients each step.

Sum a buffer across all ranks and give everyone the result. The collective that data-parallel training lives on.

Split each layer's matrices across GPUs (Megatron-style). Huge models fit, but every layer needs collectives.

Split the layer stack into stages on different GPUs and stream micro-batches through. Light traffic, but a bubble.

Shard optimizer state, gradients and parameters across data-parallel ranks instead of replicating them.

Do the math in 16- or 8-bit formats on tensor cores and keep an FP32 master copy for stable updates.

Throw away activations during forward and recompute them in backward: trade ~33% more compute for far less memory.

Attainable FLOP/s = min(peak compute, arithmetic intensity × memory bandwidth). Tells you which wall you're hitting.

Useful model FLOPs per second divided by the hardware's peak. The honest headline number for training efficiency.

Why 2× the GPUs rarely gives 2× the speed: serial work, communication and synchronization (Amdahl's law and friends).

When a warp's 32 threads read adjacent addresses, the hardware merges them into a few wide transactions.

Resident warps per SM as a fraction of the maximum: the scheduler's pool for hiding latency.

Reading, decoding and tokenizing data fast enough to keep the GPUs fed: the bottleneck after you fix everything else.

The first tool you reach for: per-GPU utilization, memory, temperature, power, clocks, processes and configuration.

Data Center GPU Manager: fleet-scale GPU telemetry, health checks, diagnostics and policy, the backbone of GPU monitoring.

Error codes the NVIDIA kernel driver writes to the system log when something goes wrong on a GPU.

HBM is protected by error-correcting codes: single-bit errors are fixed silently, double-bit errors stop the work.

In synchronous training the whole job runs at the speed of its slowest GPU. Finding that GPU is a core ops skill.

Periodically save model and optimizer state so a failure costs minutes of progress, not days.

Time-slicing, MPS and MIG: three ways to share a GPU, with very different isolation guarantees.

Unauthorized cryptocurrency mining on stolen GPU time: the most common way GPU clusters get abused.

A Jupyter server with no token on 0.0.0.0 is remote code execution for anyone who can reach it.

Container images, model weights and job scripts are code you run, and a common place for leaked credentials.

kWh, PUE and CO₂e: measuring the environmental cost of compute so it can be managed, not guessed.

Lower a GPU's power limit and give up a little speed for a lot less energy: often 20% less power for under 10% slower.

Scheduler policies that balance access between teams based on what they're entitled to and what they've recently used.