The Math Behind AI: Matrix Multiplication
See that a whole layer of neurons is one matrix multiplication, the operation GPUs are built around.
So far we’ve talked about neurons one at a time. Hardware people see them differently: as tables of numbers.
Take a layer and write its weights in a grid: one row per neuron, one column per input. That grid is a matrix. Now look at what a neuron does: multiply each input by its weight and add it all up. That is exactly “one row of the grid times the column of inputs”. Do it for every row, and you’ve computed the whole layer in one operation: matrix multiplication.
Here’s why this matters so much. Every output is computed from its own row and the shared inputs, and no output needs any other output. Four outputs or four million, they can all be worked on at the same time, by different workers. That’s the GPU’s dream job.
Real models make the grids even bigger. Instead of one input at a time, they process a batch: many examples (or many tokens) at once. The inputs become a grid too, and the whole layer becomes grid × grid. One weight row now meets many input columns, so every weight fetched from memory gets used several times, a detail that turns out to be crucial three lessons from now.
The same pattern runs through a whole transformer: the query, key and value projections, the attention output and the feed-forward layers are all matrix multiplications, and together they make up the vast majority of the work. If you make matrix multiplication fast, you make AI fast. GPU makers have spent the last decade doing exactly that.
Key takeaways
- A layer of neurons = weight matrix × input vector: one row of weights per neuron.
- With a batch of inputs, it becomes matrix × matrix.
- Every output number is independent of the others: perfect for parallel workers.
- Transformers are mostly matrix multiplications, so fast matrix maths means fast AI.
Check your understanding
Go deeper (more technical)
Multiplying an (M × K) matrix by a (K × N) matrix takes 2 · M · N · K FLOPs: M·N outputs, each a K-long dot product. In a transformer, M is the number of tokens in the batch, K the input width and N the output width.
Libraries such as cuBLAS and CUTLASS contain hand-tuned kernels for this “GEMM” (GEneral Matrix Multiply) operation. In large language models, matrix multiplications usually account for the large majority of FLOPs; attention scores, normalisation and activations make up most of the rest.