Warp & SIMT

Read it in Chapter 4 · checkpoint +50 XP →

32 threads that execute in lockstep. Divergent branches and scattered memory access cost real performance.

NVIDIA GPUs execute threads in groups of 32 called warps. This is Single Instruction, Multiple Threads (SIMT): the warp shares one instruction stream, while each thread has its own registers.

Two consequences dominate kernel performance:

1. Branch divergence: if threads in a warp take different paths of an if, the warp runs both paths with lanes masked off. Keep control flow uniform within a warp where you can.
2. Memory coalescing: when the 32 threads read adjacent addresses, the hardware merges them into a few wide transactions. Scattered access multiplies memory traffic. See Memory coalescing.

Warps are also the unit the scheduler juggles to hide latency. More resident warps (higher Occupancy) gives it more choices, up to the point where registers and shared memory run out.

Related

Sources & further reading

• CUDA C++ Programming Guide
• PRACE Best Practice Guide – GPGPU — GPU programming and tuning practice for HPC
• Modal GPU Glossary — hardware and software vocabulary

Written for LazyLab in our own words; sources are cited for depth and verification.