Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network architecture that increases model capacity without proportionally increasing computation by routing each input to a small subset of specialized 'expert' sub-networks. An MoE layer contains multiple expert networks (e.g., 8, 16, or 64 feed-forward networks) and a gating/router network that determines which experts process each token. Only the selected experts are activated for a given input, so the total compute per token remains roughly constant while the total parameter count is much larger.

For example, a model with 8 experts and top-2 routing has 8× the feed-forward parameters of a dense model but only uses 2 experts per token, so the per-token compute is only ~2× that of a single expert. This decoupling of parameters from compute allows MoE models to scale to very large parameter counts (hundreds of billions or trillions) while maintaining reasonable inference costs. The router is trained end-to-end alongside the experts, learning to specialize experts for different types of inputs or tasks.

MoE introduces several challenges: load balancing (ensuring all experts receive roughly equal training signal, often addressed with auxiliary loss), routing instability, increased memory requirements (all expert parameters must be loaded even if only a subset is used per token), and communication overhead in distributed training. Despite these challenges, MoE has been adopted in major models including Mixtral, GPT-4, and DeepSeek-V3, and is a key technique for scaling model capacity efficiently.