Mixture of Experts (MoE)
Mixture of Experts (MoE) is a neural network architecture that replaces dense feed-forward layers with multiple specialized sub-networks called “experts” and uses a learned routing mechanism to selectively activate only a subset of those experts for each input. This enables models to be massive in total parameter count while keeping per-token computation lean.