Kubernetes & the GPU Operator

Read it in Chapter 2 · checkpoint +50 XP →

Running GPU workloads on Kubernetes: device plugins, the GPU Operator, MIG profiles and gang scheduling.

Kubernetes doesn't know about GPUs by default. The NVIDIA device plugin advertises nvidia.com/gpu as a schedulable resource. The GPU Operator installs and manages the whole stack as containers: driver, container toolkit, device plugin, DCGM exporter and MIG manager.

Differences from Slurm that matter for training

• Kubernetes schedules pods one at a time. Distributed training needs gang scheduling (all-or-nothing), provided by add-ons such as Kueue or Volcano.
• Topology awareness (same rack, same NVLink domain) needs extra configuration.
• It excels at mixed fleets: inference services, notebooks and CI next to batch jobs, often on MIG slices.

Many organizations run both: Slurm for large training, Kubernetes for everything else, sometimes on the same hardware.

Related

Sources & further reading

• NVIDIA GPU Operator

Written for LazyLab in our own words; sources are cited for depth and verification.