Advanced · Neural Networks

Pruning

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Removing redundant weights or neurons from a network to reduce size and improve inference speed.

Technical Definition

Removing redundant weights or neurons from a network to reduce size and improve inference speed.

How it works

Magnitude pruning removes weights near zero. Structured pruning removes entire neurons or heads. The Lottery Ticket Hypothesis suggests dense networks contain sparse subnetworks matching full performance. Pruning can achieve 50-90% sparsity with minimal accuracy loss.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Knowledge Distillation — Compressing a large teacher model into a smaller student model by training the student to mimic the teacher's outputs.
• Quantization — Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.