Mixture of Experts (MoE): huge models where each token uses only a fraction
TL;DR
MoE splits a model into many experts and routes each token through only a few, giving large capacity at low compute. Speed follows active parameters but memory follows total parameters, the most common misunderstanding when self-hosting.
A model with hundreds of billions of parameters, yet each token only "uses" a few tens of billions, sounds contradictory. That's exactly the idea behind Mixture of Experts (MoE): instead of every parameter processing every token, the model has many "experts" and a router picks a few suitable ones per token. The model gets large capacity while compute per token stays low. Many modern LLMs use this architecture.
The mechanism
- In each Transformer layer, the feed-forward block is replaced by several experts (each a feed-forward network).
- A router (gating network) scores the experts for each token and picks the top-k.
- The output is a weighted sum of the chosen experts; the rest do no computation for that token.
token → router → [E1: 0.6] [E5: 0.4] (other experts skipped)
output = 0.6·E1(token) + 0.4·E5(token)Three milestones
| Study | Contribution |
|---|---|
| Sparsely-Gated MoE (Shazeer et al., 2017) | An MoE layer with thousands of experts and sparse activation; over 1000× more model capacity with only a small increase in computation |
| Switch Transformer (Fedus et al., 2021) | Simplified routing: each token goes to one expert; scaled to trillions of parameters with substantial pretraining speedups over dense models at equal compute |
| Mixtral 8x7B (Mistral AI, 2024) | An open model: 8 experts per layer, 2 chosen per token; 47B total parameters but only 13B active per token, matching or beating Llama 2 70B on many benchmarks |
Dense vs MoE
| Dense | MoE | |
|---|---|---|
| Parameters used per token | All | A small fraction |
| Inference speed | Proportional to total parameters | Proportional to active parameters |
| GPU memory | Holds everything | Still holds all experts |
| Training difficulty | More stable | Must balance load across experts |
Risks and caveats
- "13B active" doesn't mean it runs on a weak machine: all 47B parameters must still sit in memory. This is the most common misunderstanding when self-hosting.
- Load balancing: if the router funnels tokens to a few experts, others are wasted and the system slows down; training needs auxiliary losses to balance load.
- Experts don't specialise by topic as the name suggests: analysis of Mixtral found routing follows syntax more than subject domain.
- Fine-tuning is trickier: MoE models can be less stable to fine-tune than dense ones.
How to apply it
- Dev: When choosing an open model to self-host, size VRAM by total parameters and estimate speed by active parameters.
- Tech Lead: MoE suits high-traffic serving on large GPU clusters; on a single GPU, a small dense model is usually more practical.
- PM: When a vendor advertises "X billion parameters", ask whether that's total or active; it directly affects infrastructure cost.
Sources & references
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Shazeer et al. (arXiv)
- Switch Transformers — Fedus et al. (arXiv)
- Mixtral of Experts — Mistral AI (arXiv)