Advanced · Neural Networks

SwiGLU

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A gated linear-unit activation, x · Swish(W₁x) ⊙ (W₂x), that consistently outperforms ReLU and GELU in Transformer FFN blocks.

Technical Definition

A gated linear-unit activation, x · Swish(W₁x) ⊙ (W₂x), that consistently outperforms ReLU and GELU in Transformer FFN blocks.

How it works

SwiGLU (Shazeer, 2020) replaces the two-layer ReLU/GELU FFN with a gated variant: one projection passes through Swish (a.k.a. SiLU) and is multiplied element-wise by a second linear projection, then projected back. The gating boosts modeling quality enough that LLaMA, PaLM, and Mistral all adopted it, despite adding a third weight matrix (compensated by shrinking hidden width by 2/3).

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Activation Function — A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.
• ReLU — A simple activation function that outputs the input if positive and zero otherwise.