Advanced · Neural Networks

Mixture of Experts (MoE)

Visual diagram · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An architecture routing inputs to specialized sub-networks, activating only a subset for each input to scale efficiently.

Technical Definition

An architecture routing inputs to specialized sub-networks, activating only a subset for each input to scale efficiently.

How it works

MoE layers contain multiple parallel expert networks and a gating network. Typically top-2 experts are activated per token. This enables models with trillions of parameters while keeping inference cost manageable. Mixtral 8x7B uses this approach.

Visual Explanation (flowchart)

Input → Gating Network → [Expert 1 (selected), Expert 2 (selected), Expert 3, Expert 4] → Weighted Sum → Output

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.