Input pipeline

Read it in Chapter 5 · checkpoint +50 XP →

Reading, decoding and tokenizing data fast enough to keep the GPUs fed: the bottleneck after you fix everything else.

Once the model math is fast, the CPU side often becomes the limit: reading shards from storage, decompressing, tokenizing, shuffling and collating batches.

Symptoms: GPU utilization saw-tooths, step time doesn't improve when you optimize the model, and logs show time waiting on the next batch.

Fixes

• More data-loader workers (processes) per rank, and pinned memory.
• Prefetching: prepare batch k+1 while batch k trains (see CUDA streams & async execution).
• Pre-tokenize offline and store packed binary shards. It's the single biggest win for LLM pretraining.
• Make sure the parallel filesystem or object store can deliver the aggregate bandwidth: GPUs × tokens/s × bytes per token, plus checkpoint traffic.

The Feed the Beast mission shows the bottleneck moving to data loading the moment communication is fixed.

Use it in a mission

Related

Sources & further reading

• Distributed Training (T. Tsugriy) — parallelism strategies, collectives, memory

Written for LazyLab in our own words; sources are cited for depth and verification.