Advanced · Neural Networks

Sparse Mixture of Experts (SMoE)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A neural network architecture where a 'router' network activates only a few specialized 'expert' subnetworks for each input, improving efficiency.

Technical Definition

A neural network architecture where a 'router' network activates only a few specialized 'expert' subnetworks for each input, improving efficiency.

How it works

SMoEs allow models to scale to vastly larger numbers of parameters without a proportional increase in computational cost during inference. For each input, only a subset of the expert networks are engaged, making the model computationally efficient while allowing it to specialize across different types of data or tasks. This architecture is key to building very large yet manageable LLMs.

Related Concepts

• Mixture of Experts (MoE) — An architecture routing inputs to specialized sub-networks, activating only a subset for each input to scale efficiently.