Advanced · Safety

Interpretability

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Understanding why a model makes the predictions it does, by inspecting its internals.

Technical Definition

Understanding why a model makes the predictions it does, by inspecting its internals.

How it works

Interpretability ranges from feature attributions (SHAP, integrated gradients) to mechanistic interpretability (circuits, sparse autoencoders) that reverse-engineers neural networks. It is crucial for debugging, scientific understanding, and trust — and is often considered a prerequisite for safely deploying advanced AI.

Related Concepts

• Explainability (XAI) — Techniques making AI decisions understandable to humans, crucial for trust and regulatory compliance.
• AI Safety & Alignment — The field ensuring AI systems behave as intended, remain under human control, and avoid unintended harm.
• AI Alignment — The research field aimed at making AI systems pursue the goals their developers and users actually intend.