Beginner · Neural Networks

ReLU

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A simple activation function that outputs the input if positive and zero otherwise.

Technical Definition

A simple activation function that outputs the input if positive and zero otherwise.

How it works

ReLU(x) = max(0, x). It became the default activation in deep networks because it is cheap to compute, doesn't saturate for positive inputs (avoiding vanishing gradients), and induces sparsity. Variants like Leaky ReLU and GELU address the 'dying ReLU' problem where neurons get stuck at zero.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Activation Function — A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.
• Deep Learning — A subset of machine learning using neural networks with many layers to learn hierarchical representations from large datasets.