Advanced · Neural Networks

Mamba

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A selective state-space model (SSM) architecture that processes sequences in linear time with content-aware gating, rivaling Transformers on language tasks.

Technical Definition

A selective state-space model (SSM) architecture that processes sequences in linear time with content-aware gating, rivaling Transformers on language tasks.

How it works

Mamba (Gu & Dao, 2023) extends structured state-space models with input-dependent parameters, letting the model selectively remember or forget information. Unlike Transformer self-attention which is O(n²), Mamba runs in O(n) with constant memory per step, enabling very long contexts. It uses a hardware-aware parallel scan on GPUs. Hybrid Mamba-Transformer models (Jamba, Zamba) combine the strengths of both.

Mathematical Notation

h_t = A(x_t) h_{t-1} + B(x_t) x_t,  y_t = C h_t

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• Long Context — The ability of an LLM to process and maintain coherence over very long input sequences or conversations, beyond typical token limits.