Intermediate · Neural Networks

Recurrent Neural Network (RNN)

Visual diagram · Math · Worked example · 3 difficulty levels.

TL;DR. A neural network with loops that maintain hidden state, designed to process sequential data like text and time series.

Technical Definition

A neural network with loops that maintain hidden state, designed to process sequential data like text and time series.

How it works

RNNs process sequences one element at a time, maintaining a hidden state that captures information from previous steps. However, vanilla RNNs suffer from vanishing/exploding gradients. LSTM and GRU address this with gating mechanisms. While largely superseded by Transformers for NLP, RNNs remain useful for streaming and low-latency applications.

Mathematical Notation

hₜ = tanh(Wₕhₜ₋₁ + Wₓxₜ + b)

The hidden state hₜ is computed from the previous hidden state hₜ₋₁ and current input xₜ, transformed by weight matrices and a tanh activation.

Visual Explanation (flowchart)

x₁ → RNN Cell → h₁ → x₂ → RNN Cell → h₂ → ... → xₙ → RNN Cell → hₙ → Output

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Activation Function — A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.