Intermediate · Research
Distillation
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. Compressing a large AI model into a smaller one that mimics the original's performance, improving efficiency.
Technical Definition
Compressing a large AI model into a smaller one that mimics the original's performance, improving efficiency.
How it works
Knowledge distillation is a technique where a smaller, more efficient 'student' model is trained to replicate the behavior of a larger, more complex 'teacher' model. The goal is to transfer the knowledge from the teacher to the student, allowing for faster inference and reduced computational resources while retaining a significant portion of the original model's accuracy.
Related Concepts
- Transfer Learning — Leveraging knowledge from a model trained on one task to improve performance on a different but related task.
- Model Compression — Techniques for shrinking models while preserving most of their performance.