Intermediate · Neural Networks

Residual Connection (Skip Connection)

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A shortcut that adds a layer's input directly to its output, enabling training of very deep networks.

Technical Definition

A shortcut that adds a layer's input directly to its output, enabling training of very deep networks.

How it works

Residual connections add the input directly to the output: y = F(x) + x. Layers only learn the residual from identity. They solve the degradation problem and provide gradient highways during backpropagation. Used extensively in Transformers and ResNets.

Mathematical Notation

y = F(x, {Wᵢ}) + x

If the optimal transformation is close to identity, the network only needs to learn a small residual.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
• Deep Learning — A subset of machine learning using neural networks with many layers to learn hierarchical representations from large datasets.

Further Reading

• ResNet Paper