Intermediate · Data

Dataset Shift

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A change in the distribution of the data between the training and test (or deployment) phases.

Technical Definition

A change in the distribution of the data between the training and test (or deployment) phases.

How it works

Dataset shift broadly describes any change in data distribution from training to deployment, causing models to perform poorly. This can manifest as 'covariate shift' (input features change), 'concept drift' (input-output relationship changes), or 'prior probability shift' (class proportions change). Detecting and mitigating dataset shift is crucial for maintaining model performance in real-world dynamic environments, often requiring model retraining or adaptation.