Intermediate · Data
Information gain
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A metric used in decision trees to measure the expected reduction in entropy achieved by splitting data on a particular feature.
Technical Definition
A metric used in decision trees to measure the expected reduction in entropy achieved by splitting data on a particular feature.
How it works
Information gain quantifies the usefulness of a feature for splitting data in a decision tree or forest. It's calculated as the difference between the entropy of a node (representing impurity or disorder) and the weighted average entropy of its child nodes after the split. Features with higher information gain are considered more effective at partitioning the data.
Related Concepts
- Feature Selection — Choosing the most useful subset of features to improve performance and interpretability.
- Decision tree — A decision tree is a supervised learning model that uses a tree-like structure of decisions and their possible consequences.
- Entropy — A measure of the unpredictability or randomness in a probability distribution.