Beginner · Data

Data Cleaning

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Identifying and fixing or removing errors, inconsistencies, and duplicates in a dataset.

Technical Definition

Identifying and fixing or removing errors, inconsistencies, and duplicates in a dataset.

How it works

Common cleaning tasks: handling missing values, removing duplicates, correcting typos, normalizing units, deduplicating near-identical text. ML practitioners often spend the majority of their time cleaning data — the unglamorous prerequisite to every successful model.

Related Concepts

• Dataset — An organized collection of examples used to train, validate, or test a model.
• Data Pipeline — An automated workflow that ingests, transforms, and delivers data to models.
• Data Preprocessing — Transforming raw data into a format suitable for model training.