Securing Shared GPUs

6.1Containers & GPUs

A container image carries the CUDA runtime and libraries, but the GPU driver must come from the host (it's a kernel module). The NVIDIA Container Toolkit bridges the two. At start it mounts the driver's user-space libraries and the /dev/nvidia* device nodes into the container.

Practical rules

• Pin images to a CUDA version the host driver supports (see The CUDA software stack).
• HPC sites often use Enroot/Pyxis or Apptainer with Slurm. They run images unprivileged, which is a big security win.
• Never run training containers --privileged. Device access should be granted narrowly by the runtime.

Security angle: the device nodes are the GPU. Anything that can open /dev/nvidia* outside the scheduler's control can use the GPU. That's exactly how an unauthorized miner hides in plain sight. See Cryptojacking on GPU clusters and Multi-tenancy & GPU sharing.

6.2Multi-tenancy & GPU sharing

Shared clusters must decide how tenants share GPUs:

Enforcement matters more than policy. The scheduler should be the only path to a GPU: cgroup device controls restrict /dev/nvidia* to processes inside a job allocation. Without that, any user with a shell on a node can quietly use idle GPUs. See Cryptojacking on GPU clusters.

Pair it with least-privilege accounts, no shared service users, audit logs and per-tenant accounting so usage is visible and fair (see Fair-share scheduling).

6.3Multi-Instance GPU (MIG)

MIG (A100, H100 and later) splits a GPU into up to 7 instances. Each gets a dedicated slice of SMs, L2 cache and HBM, with its own memory controllers. Faults and memory pressure in one instance don't leak into another, which makes MIG the strongest of the GPU-sharing options. See Multi-tenancy & GPU sharing.

Profiles are named by compute slices and memory, e.g. on an 80 GB H100: 1g.10gb, 2g.20gb, 3g.40gb, 4g.40gb, 7g.80gb. Not every combination fits. Placement follows fixed rules about which slices can sit together.

When to use it: inference, notebooks and CI workloads that need a fraction of a GPU and must not interfere with each other. When not to: large training jobs. They want whole GPUs and NVLink, which MIG instances can't use for peer-to-peer.

Play with the partitioner below to see how profiles pack.

6.4Exposed notebooks & dashboards

Jupyter is designed to run arbitrary code: that's the point. With --ip=0.0.0.0 and authentication disabled (empty token and password), anyone who can reach the port can run commands as the notebook's user. The same applies to many ML dashboards, experiment trackers and inference UIs.

Safer patterns

• Bind to 127.0.0.1 and reach it through an SSH tunnel or an authenticated gateway (JupyterHub with SSO).
• Never disable tokens or passwords. Use TLS when traffic leaves the host.
• Firewall compute nodes: only the scheduler, monitoring and admin networks should reach them.
• Run notebooks inside scheduler allocations so they're time-limited and accounted.

Detecting: ss -tlnp shows listeners on 0.0.0.0. Server logs record requests from unexpected IPs. Network monitoring flags inbound connections to compute nodes from outside.

$ ss -tlnp
LISTEN 0 128 0.0.0.0:8888 0.0.0.0:* users:(("python3",pid=2741))

6.5Cryptojacking on GPU clusters

GPUs are valuable, and attackers who get code execution on a GPU node very often just mine cryptocurrency. Common entry points:

• Exposed, unauthenticated services: notebooks, dashboards and ML tooling UIs on 0.0.0.0 (see Exposed notebooks & dashboards).
• Leaked credentials and API keys in images, repos or job scripts (see Supply chain & secrets).
• Over-privileged containers.

Detection signals

• A GPU busy with no scheduled job: compare DCGM utilization to scheduler state.
• Unknown process names on GPUs (nvidia-smi), often disguised as kernel threads.
• Outbound connections to mining pools (stratum protocol, ports like 3333/4444) in ss -tnp or firewall logs.
• New cron entries or systemd units owned by unexpected users.

Response: preserve evidence → kill the process → remove persistence → close the entry point → lock or rotate the affected identities → look for lateral movement → report. Killing the process alone is never enough. The Night Shift mission makes that concrete.

6.6Supply chain & secrets

A training job pulls in a lot of other people's code: base images, Python packages, custom kernels, pretrained weights and datasets. Each is part of your supply chain.

Risks

• Images with baked-in secrets (cloud keys, tokens) are readable by anyone who can pull them.
• Pickle-based model files can execute code when loaded. Prefer safetensors.
• Unpinned dependencies change underneath you.

Controls

• Pin images by digest, scan them, and build from trusted bases.
• Inject secrets at runtime from a secret manager, scoped per job. Never put them in #SBATCH scripts, env files in repos or image layers.
• Short-lived credentials and least privilege for storage access.
• Verify checksums or signatures for weights and datasets.

These controls protect your cluster and also the people whose data and models you're responsible for.