CUDA streams & async execution
Read it in Chapter 4 · checkpoint +50 XP →
Queues of GPU work that let copies, kernels and communication overlap instead of waiting on each other.
A stream is an ordered queue of GPU operations. Work in different streams can run concurrently: a host-to-device copy in one, a kernel in another, an NCCL collective in a third. Events express dependencies between streams without blocking the CPU.
This is the mechanism behind every "overlap" optimization:
- Gradient all-reduce overlapping with the backward pass in DDP uses a separate communication stream.
- Prefetching the next batch while the current one computes (see Input pipeline).
- ZeRO-3/FSDP prefetching the next layer's parameters (see ZeRO & FSDP).
Projects such as grCUDA go further. They build the dependency graph from the data each kernel touches and schedule independent kernels on separate streams automatically, so polyglot programs get concurrency without hand-written stream code.
Pitfall: any implicit synchronization (cudaDeviceSynchronize, .item() in PyTorch, pageable memory copies) collapses your carefully built overlap back into serial execution.
Related
Sources & further reading
- NVIDIA grCUDA — polyglot GPU access and asynchronous kernel scheduling
- CUDA C++ Programming Guide
- CUDA C++ Best Practices Guide
Written for LazyLab in our own words; sources are cited for depth and verification.