Advanced · Neural Networks

Long Short-Term Memory (LSTM)

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. An RNN variant with gating mechanisms that can learn long-range dependencies without suffering from vanishing gradients.

Technical Definition

An RNN variant with gating mechanisms that can learn long-range dependencies without suffering from vanishing gradients.

How it works

LSTMs solve the vanishing gradient problem by introducing a cell state with three gates: forget gate (what to discard), input gate (what new info to store), and output gate (what to expose). This enables learning dependencies across hundreds of timesteps.

Mathematical Notation

fₜ = σ(Wf·[hₜ₋₁, xₜ] + bf)    iₜ = σ(Wi·[hₜ₋₁, xₜ] + bi)

The forget gate fₜ and input gate iₜ use sigmoid to produce values between 0 and 1, controlling information flow through the cell state.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Recurrent Neural Network (RNN) — A neural network with loops that maintain hidden state, designed to process sequential data like text and time series.