Home › Glossary › Neural Networks › Dropout

Beginner · Neural Networks

Dropout

Visual diagram · Math · Worked example · 3 difficulty levels.

TL;DR. Dropout randomly turns off neurons during training to prevent the network from relying too heavily on any single neuron, improving generalization.

Intuitive Explanation

Imagine a basketball team where the coach randomly benches different players each practice. Every player must learn to play well regardless of who else is on the court. This makes the whole team more resilient. Dropout does the same — every neuron must learn to be useful independently.

Technical Definition

Dropout is a stochastic regularization technique that, during training, independently sets each neuron's output to zero with probability p. At inference, all neurons are active but outputs are scaled by (1-p). This approximates an ensemble of 2^n sub-networks, providing regularization through implicit model averaging.

How it works

Dropout (Srivastava et al., 2014) is one of the simplest yet most effective regularization techniques. **How It Works:** - During training: each neuron is randomly "dropped" (set to zero) with probability p - During inference: all neurons are active, outputs scaled by (1-p) to maintain expected values - Effectively trains an exponential number of sub-networks simultaneously **Why It Works (Multiple Theories):** 1. **Ensemble Effect:** Each training step uses a different sub-network. The final model averages their predictions. 2. **Co-adaptation Prevention:** Neurons can't rely on specific other neurons being present, forcing redundant representations. 3. **Noise Injection:** Acts as a form of data augmentation in the hidden layers. 4. **Bayesian Interpretation:** Approximate Bayesian inference through stochastic weight averaging. **Practical Guidelines:** - p=0.5 for hidden layers, p=0.2 for input layers (original paper) - Modern practice: p=0.1-0.3 for Transformers, p=0.2-0.5 for MLPs - Don't use dropout with BatchNorm — they interact poorly - In modern architectures, dropout is often replaced by other regularizers **Variants:** - **Spatial Dropout (2D):** Drops entire feature maps in CNNs - **DropConnect:** Drops individual weights instead of neurons - **DropPath/Stochastic Depth:** Drops entire residual blocks

Mathematical Notation

hᵢ' = hᵢ · mᵢ    where mᵢ ~ Bernoulli(p)

Each neuron activation hᵢ is multiplied by a random mask mᵢ that is 1 with probability p (keep) and 0 with probability 1-p (drop). During inference, no mask is applied but activations are scaled by p.

Visual Explanation (layers)

Full Network (all neurons active) → Dropout Applied (random neurons zeroed) → Training Step → Different Dropout Pattern → Next Step

Real-World Use Cases

  • Google: Original dropout paper from Hinton's group at Google (now standard in most architectures)
  • OpenAI: GPT models use attention and residual dropout for regularization
  • Meta: DropPath (stochastic depth) is used in Vision Transformers

Related Concepts

  • Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
  • Overfitting — When a model learns noise and specific patterns in training data too well, causing it to perform poorly on new, unseen data.
  • Regularization — Techniques that constrain a model's complexity to prevent overfitting and improve generalization to unseen data.

Further Reading

  • Dropout Paper — Srivastava et al.