Advanced · Neural Networks

Knowledge Distillation

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. Compressing a large teacher model into a smaller student model by training the student to mimic the teacher's outputs.

Technical Definition

Compressing a large teacher model into a smaller student model by training the student to mimic the teacher's outputs.

How it works

The student learns from the teacher's soft probability outputs, which contain richer information about inter-class relationships than hard labels. The temperature parameter softens outputs to reveal more nuance. Applications include mobile deployment and reducing inference costs.

Mathematical Notation

L = α·CE(y, σ(zs)) + (1-α)·KL(σ(zt/T), σ(zs/T))·T²

Combines standard cross-entropy with KL divergence between teacher and student soft outputs at temperature T.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• Softmax Function — A function that converts a vector of raw scores into a probability distribution where all values sum to one.