Inside the Transformer: The Engine Behind Modern LLMs
See the whole machine: how tokens flow up a tall stack of identical blocks and come out as predictions.
You’ve now met every major part. Time to bolt them together.
A transformer is surprisingly regular. Tokens become embeddings (plus position information), then pass up through a stack of identical blocks. Each block does two things:
- Attention: tokens look at each other and gather context, with many heads in parallel.
- Feed-forward: each token is then processed on its own by two big layers of neurons, just like the hidden layers from Track 1. About two-thirds of the weights live here, and research suggests much of the model’s factual knowledge does too.
The blue line in the picture is the secret to stacking dozens of blocks. A block doesn’t replace a token’s vector; it adds its contribution. Researchers call this running vector the residual stream: a shared notepad every block reads from and writes to. Early blocks tend to jot down simple things (which word is this, what’s next to it); later ones add more abstract notes (who “it” refers to, what the sentence is about).
After the last block, the final token’s vector is compared with every entry in the vocabulary, giving one score per token, and softmax turns those into the next-token probabilities you met in the very first lesson.
That’s the whole machine. GPT-2, Llama and today’s giant models share the blueprint; what changes is how tall the stack is and how wide the vectors are, which is exactly what the simulation lets you play with.
Key takeaways
- A transformer = embeddings → many identical blocks → probabilities for the next token.
- Each block: attention (tokens talk to each other) + feed-forward (each token thinks on its own).
- Blocks add to a shared residual stream instead of overwriting it, which makes deep stacks trainable.
- Parameters grow with the number of blocks and with the width squared; bigger models are the same design, scaled.
Check your understanding
Go deeper (more technical)
Per block: attention has 4 matrices of width × width (query, key, value, output) = 4d² weights; the feed-forward network expands to 4d and back = 8d². Hence ≈ 12d² per block, two-thirds of it feed-forward. Small extras (biases, normalisation) are ignored here.
Besides the residual stream, layer normalisation keeps the numbers in a stable range as they flow up the stack. Studies such as Geva and colleagues (2021) found feed-forward layers act like key–value memories that store associations, which is why researchers editing facts inside models often target them.
Modern models tweak the recipe (RMSNorm, SwiGLU feed-forward, rotary positions, grouped-query attention) but the overall shape is unchanged since 2017. Llama 3 8B: 32 blocks, width 4,096; Llama 3 70B: 80 blocks, width 8,192.