Slurm

Read it in Chapter 2 · checkpoint +50 XP →

The batch scheduler that runs most HPC and many AI clusters: partitions, jobs, allocations and node states.

Slurm turns a pile of servers into a shared machine. Key concepts:

• Partitions: named groups of nodes with limits (sinfo lists them).
• Jobs are submitted with sbatch script.sh. #SBATCH lines in the script request resources: --nodes, --gpus-per-node, --time, --partition.
• Allocation: the scheduler picks nodes (backfill, priority, Fair-share scheduling) and runs the script on the first one. srun launches tasks across the rest.
• Node states: idle, alloc, drain (finish current work, take no new work) and down.

Daily operator commands

squeue --me                       # what's running / pending and why
sacct                             # history with exit codes
scontrol show job <id>            # everything about a job
scontrol update NodeName=node-3 State=DRAIN Reason="gpu5 fan"
sinfo -R                          # why nodes are drained
scontrol requeue <id>             # restart a job (from its checkpoint)

Draining with a clear reason is the polite way to say "this hardware is suspect" without killing anyone's job.

Use it in a mission

Related

Sources & further reading

• Slurm documentation

Written for LazyLab in our own words; sources are cited for depth and verification.