Beginner · Neural Networks

Activation Function

Visual diagram · Math · Worked example · 3 difficulty levels.

TL;DR. A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.

Technical Definition

A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.

How it works

Activation functions introduce non-linearity into neural networks. Without them, any number of stacked layers would collapse into a single linear transformation. Common choices include ReLU (Rectified Linear Unit), which outputs max(0, x); Sigmoid, which squashes values to (0, 1); and Tanh, which maps to (-1, 1). Modern architectures often use GELU or SiLU for smoother gradients.

Mathematical Notation

ReLU(x) = max(0, x)    Sigmoid(x) = 1 / (1 + e⁻ˣ)

ReLU is the most widely used: it passes positive values unchanged and zeroes out negatives. Sigmoid compresses any input into a probability-like range between 0 and 1.

Visual Explanation (flowchart)

Input x → [ReLU: slope 0 for x<0, slope 1 for x≥0] → Output | [Sigmoid: S-curve from 0 to 1]

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
• Deep Learning — A subset of machine learning using neural networks with many layers to learn hierarchical representations from large datasets.