Running Jobs

2.1The CUDA software stack

From the bottom up:

1. Kernel module (nvidia.ko and friends). It owns the hardware, memory management (including Unified Memory / UVM), interrupts and error reporting. NVIDIA publishes an open-source version, open-gpu-kernel-modules, the default for Turing and newer GPUs. The Xid messages you read in dmesg come from here.
2. User-mode driver (libcuda.so): the CUDA driver API, shipped with the driver.
3. CUDA runtime and toolkit (libcudart, nvcc): what applications compile against.
4. Libraries: cuBLAS, cuDNN, NCCL, CUTLASS.
5. Frameworks: PyTorch, JAX, plus higher-level tooling.

Compatibility rule: a newer driver runs older CUDA runtimes, but not the reverse. nvidia-smi prints the driver version and the maximum CUDA version it supports. Containers bundle their own runtime but must use the host's driver, injected by the NVIDIA Container Toolkit. See Containers & GPUs.

2.2Slurm

Slurm turns a pile of servers into a shared machine. Key concepts:

• Partitions: named groups of nodes with limits (sinfo lists them).
• Jobs are submitted with sbatch script.sh. #SBATCH lines in the script request resources: --nodes, --gpus-per-node, --time, --partition.
• Allocation: the scheduler picks nodes (backfill, priority, Fair-share scheduling) and runs the script on the first one. srun launches tasks across the rest.
• Node states: idle, alloc, drain (finish current work, take no new work) and down.

Daily operator commands

squeue --me                       # what's running / pending and why
sacct                             # history with exit codes
scontrol show job <id>            # everything about a job
scontrol update NodeName=node-3 State=DRAIN Reason="gpu5 fan"
sinfo -R                          # why nodes are drained
scontrol requeue <id>             # restart a job (from its checkpoint)

Draining with a clear reason is the polite way to say "this hardware is suspect" without killing anyone's job.

$ sbatch train.sh
sbatch: error: Batch job submission failed: Requested node configuration is not available

2.3NCCL

NCCL ("nickel") implements the collectives distributed training runs on: all-reduce, all-gather, reduce-scatter, broadcast and send/recv. It discovers the topology (NVLink, NVSwitch, PCIe, NICs) and builds rings and trees to use it. See All-reduce (ring & tree).

Environment variables worth knowing

• NCCL_DEBUG=INFO prints the chosen transport and channels. Read it.
• NCCL_IB_DISABLE=1 forces TCP sockets instead of InfiniBand. It's useful on laptops and a disaster in production.
• NCCL_SOCKET_IFNAME / NCCL_IB_HCA select interfaces.

Diagnosing

• all_reduce_perf from nccl-tests measures achievable bus bandwidth. Run it on idle nodes to catch bad links before users do.
• Hung collectives end with a PyTorch watchdog timeout. The cause is usually one bad rank: a dead link, a lost GPU or a straggler that stopped responding.

2.4Checkpointing & fault tolerance

At thousands of GPUs, something fails every few hours. Checkpointing bounds the damage: on failure, you restart from the last saved state and lose at most one checkpoint interval.

Choosing the interval: too frequent wastes I/O time, too rare wastes compute on failure. A classic approximation (Young/Daly) is interval ≈ √(2 × checkpoint_cost × MTBF).

Making it cheap

• Shard checkpoints: each rank writes its own slice in parallel.
• Async checkpointing: copy to host memory, then write in the background.
• Keep a few recent checkpoints. Corrupt or partial saves happen.

Ops integration: scontrol requeue or a job-level retry loop, plus a warm spare pool, turns "a GPU died" into a 5-minute blip. That's the loop you practice in Fallen Off the Bus and The Slow Lane.

2.5Kubernetes & the GPU Operator

Kubernetes doesn't know about GPUs by default. The NVIDIA device plugin advertises nvidia.com/gpu as a schedulable resource. The GPU Operator installs and manages the whole stack as containers: driver, container toolkit, device plugin, DCGM exporter and MIG manager.

Differences from Slurm that matter for training

• Kubernetes schedules pods one at a time. Distributed training needs gang scheduling (all-or-nothing), provided by add-ons such as Kueue or Volcano.
• Topology awareness (same rack, same NVLink domain) needs extra configuration.
• It excels at mixed fleets: inference services, notebooks and CI next to batch jobs, often on MIG slices.

Many organizations run both: Slurm for large training, Kubernetes for everything else, sometimes on the same hardware.

2.6Fair-share scheduling

On a shared cluster, "first come, first served" lets one team's 1,000 queued jobs starve everyone else. Fair-share scheduling (e.g. Slurm's multifactor priority) lowers the priority of users and accounts that recently used more than their share, with usage decaying over time.

The ingredients

• Shares: each account's entitlement, often matching funding or team size.
• Usage decay: a half-life (e.g. 7 days) so past usage is forgiven gradually.
• QOS limits: caps on concurrent GPUs, wall time or job count, so no single job blocks the cluster.
• Preemption: low-priority or opportunistic jobs can be checkpointed and requeued when priority work arrives. This only works if jobs checkpoint well.

Fairness is also about transparency: publish usage per team, queue wait times and energy, so allocation debates use data instead of anecdotes.