Mixture of Experts: How AI Activates Only Part of a Model
See how mixture-of-experts models store many parameters but use only a few of them for each token.
A dense model uses every weight for every token. When “the” comes in, all 70 billion parameters of a 70B model wake up and do their sums, exactly as they would for a tricky line of code. That’s a lot of effort for “the”.
A mixture of experts model (MoE) splits the feed-forward part of each block into many separate networks called experts, from 8 to a few hundred of them. In front sits a small router: a learned layer that looks at each token and picks the few experts that should handle it, often just two. The others sit idle for that token.
The result is a model with a big brain and a small appetite. Mixtral 8x7B (2023) stores 46.7 billion parameters but uses only about 12.9 billion per token, so it runs at roughly the speed of a 13B model while performing more like a much bigger one. DeepSeek-V3 (late 2024) pushes it further: 671 billion parameters, 37 billion active per token.
Don’t picture neat specialists, one for French and one for maths. Nobody assigns topics; the experts’ roles emerge in training and tend to be fuzzy, often specialising more in kinds of tokens (punctuation, numbers, bits of words) than in subjects. Training also needs care, so the router doesn’t send everything to a few favourites while the rest learn nothing.
And there’s no free lunch. Every expert must sit in GPU memory, ready in case the router picks it, so an MoE needs the memory of its full size even though each token uses only a fraction. MoE is a trade: much cheaper compute per token, the same hefty memory bill.
Key takeaways
- In an MoE model, each block has many expert networks, and a router picks a few for each token.
- Only the chosen experts compute, so the work per token is a fraction of the model’s total size.
- Expert roles are learned and fuzzy, not neat topics like “maths” or “French”.
- All experts must stay in memory: MoE saves compute, not memory.
Check your understanding
Go deeper (more technical)
Sparse mixture-of-experts layers in modern form come from Shazeer and colleagues (2017); the Switch Transformer (2021) showed that even one expert per token works. A load-balancing term in the loss nudges the router to spread tokens evenly. DeepSeek-V3 adds a “shared” expert that every token uses, plus many small routed ones.
Why isn’t “8x7B” 56 billion parameters? Only the feed-forward parts are multiplied by eight; attention and embeddings are shared. On big systems, experts are often spread across many GPUs (expert parallelism), which Track 4 covers.