Tensor Cores: Hardware Built for AI
See how tensor cores multiply whole blocks of numbers in one step, and why smaller numbers make them faster.
Once it became clear that AI is mostly matrix multiplication, GPU designers asked a bold question: why make a general-purpose core do it one multiply-add at a time?
A regular GPU core’s basic move is the fused multiply-add: take a, b and c, and produce a × b + c in one tick of the clock. A matrix multiplication is millions of those, done one by one, each core fetching its numbers and storing its result.
A tensor core works at a bigger scale. Its basic move is a whole small matrix multiply: take two little blocks of numbers, multiply them, add the result to a third block, all in one go. NVIDIA introduced them in 2017 with the V100, and every generation since has made them bigger, faster and fluent in smaller number formats. Google’s TPUs, built only for AI, take the same idea even further with one giant matrix unit.
The payoff is dramatic. On an H100, the regular cores manage about 67 trillion FP32 operations a second. The tensor cores manage about 989 trillion in BF16 (roughly fifteen times more) and about 1,979 trillion in FP8. That’s where the previous lesson pays off: every halving of the bits roughly doubles tensor-core throughput, because smaller numbers mean simpler multipliers, and more of them fit on the chip.
There’s a catch, and you know it already. Tensor cores make the maths so fast that the memory wall looms even larger. Feeding them needs everything from the last few lessons: big batches, careful tiling and compact number formats. A tensor core waiting for data is just an expensive way to wait.
Key takeaways
- A regular core does one multiply-add per tick; a tensor core multiplies whole small matrix blocks per tick.
- On an H100, tensor cores deliver about 15× the regular cores’ throughput in BF16, and about 30× in FP8.
- Halving the bits roughly doubles tensor-core speed.
- Faster maths makes the memory wall worse, so batching, tiling and small formats matter even more.
Check your understanding
Go deeper (more technical)
A single tensor-core instruction is issued by a whole warp and works on fixed tile shapes that vary by generation, for example 16 × 8 × 16 for BF16 on recent NVIDIA GPUs. Results usually accumulate in FP32 even when the inputs are BF16 or FP8, which keeps long sums accurate.
TF32 is a tensor-core format with FP32’s range but only 10 mantissa bits; many frameworks use it automatically for FP32 maths. Recent GPUs also support 2:4 sparsity (two of every four weights set to zero, for up to twice the throughput) and 4-bit floating-point formats with block scales (Blackwell’s FP4).
H100 SXM spec-sheet peaks, dense: FP32 ≈ 67, TF32 ≈ 494, BF16 ≈ 989, FP8 ≈ 1,979 TFLOP/s.