Containers & GPUs
Read it in Chapter 6 · checkpoint +50 XP →
How containers get GPU access: the host driver, injected device nodes and the NVIDIA Container Toolkit.
A container image carries the CUDA runtime and libraries, but the GPU driver must come from the host (it's a kernel module). The NVIDIA Container Toolkit bridges the two. At start it mounts the driver's user-space libraries and the /dev/nvidia* device nodes into the container.
Practical rules
- Pin images to a CUDA version the host driver supports (see The CUDA software stack).
- HPC sites often use Enroot/Pyxis or Apptainer with Slurm. They run images unprivileged, which is a big security win.
- Never run training containers
--privileged. Device access should be granted narrowly by the runtime.
Security angle: the device nodes are the GPU. Anything that can open /dev/nvidia* outside the scheduler's control can use the GPU. That's exactly how an unauthorized miner hides in plain sight. See Cryptojacking on GPU clusters and Multi-tenancy & GPU sharing.
Related
Sources & further reading
Written for LazyLab in our own words; sources are cited for depth and verification.