Occupancy

Read it in Chapter 4 · checkpoint +50 XP →

Resident warps per SM as a fraction of the maximum: the scheduler's pool for hiding latency.

Each SM can host a limited number of warps (64 on H100). Occupancy is how many are actually resident, limited by each block's use of:

• Registers per thread (the register file is shared by all resident warps),
• Shared memory per block,
• Threads per block and blocks per SM.

Higher occupancy gives the warp scheduler more candidates to switch to while others wait on memory. But it's a means, not a goal. Many fast kernels (big GEMMs, FlashAttention) run at low occupancy, using registers and shared memory for data reuse instead.

Practical approach (from CUDA and PRACE best practice): use the occupancy calculator or Nsight Compute to see the limiting resource, try a few block sizes (multiples of 32), and measure. Don't assume.

Related

Sources & further reading

• CUDA C++ Best Practices Guide
• PRACE Best Practice Guide – GPGPU — GPU programming and tuning practice for HPC
• Modal GPU Glossary — hardware and software vocabulary

Written for LazyLab in our own words; sources are cited for depth and verification.