Intermediate · Neural Networks

Catastrophic Forgetting

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The phenomenon where a neural network rapidly loses previously learned information upon learning new, distinct tasks.

Technical Definition

The phenomenon where a neural network rapidly loses previously learned information upon learning new, distinct tasks.

How it works

Also known as 'catastrophic interference,' this issue arises when training a model sequentially on multiple tasks. The weights adapted for a new task overwrite those crucial for older tasks, leading to a significant degradation in performance on previously mastered skills. Mitigating catastrophic forgetting is a key challenge in continual learning research, aiming for systems that can incrementally acquire new knowledge.