Tensor Core

Read it in Chapter 4 · checkpoint +50 XP →

Matrix-multiply-accumulate units that deliver most of a modern GPU's FLOPs, if you feed them the right precision and shapes.

Tensor cores compute small matrix products (e.g. 16×16 tiles) in one instruction. On an H100 they deliver ~989 dense BF16 TFLOP/s versus ~67 TFLOP/s of plain FP32. That gap is why Mixed precision (BF16, FP8) training is the default.

Getting the FLOPs

• Use BF16/FP16 (or FP8 with Transformer Engine) for matmuls. FP32 without TF32 never touches them.
• Keep GEMM dimensions large and multiples of 8/16. Tiny micro-batches or very narrow tensor-parallel shards make small GEMMs that can't saturate the units.
• Keep data flowing from HBM & the memory hierarchy. Tensor cores are so fast that many ops (normalization, activations, attention softmax) become memory-bound. Kernel fusion such as FlashAttention exists for this.

Model FLOPs Utilization (MFU (Model FLOPs Utilization)) is ultimately a measure of how busy the tensor cores are.

Related

Sources & further reading

• NVIDIA H100 Tensor Core GPU
• Modal GPU Glossary — hardware and software vocabulary
• Mixed Precision Training (Micikevicius et al., 2018)

Written for LazyLab in our own words; sources are cited for depth and verification.