Supply chain & secrets

Read it in Chapter 6 · checkpoint +50 XP →

Container images, model weights and job scripts are code you run, and a common place for leaked credentials.

A training job pulls in a lot of other people's code: base images, Python packages, custom kernels, pretrained weights and datasets. Each is part of your supply chain.

Risks

• Images with baked-in secrets (cloud keys, tokens) are readable by anyone who can pull them.
• Pickle-based model files can execute code when loaded. Prefer safetensors.
• Unpinned dependencies change underneath you.

Controls

• Pin images by digest, scan them, and build from trusted bases.
• Inject secrets at runtime from a secret manager, scoped per job. Never put them in #SBATCH scripts, env files in repos or image layers.
• Short-lived credentials and least privilege for storage access.
• Verify checksums or signatures for weights and datasets.

These controls protect your cluster and also the people whose data and models you're responsible for.

Related

Sources & further reading

• NVIDIA Container Toolkit

Written for LazyLab in our own words; sources are cited for depth and verification.