Advanced · Safety

Activation Patching

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An interpretability technique that swaps activations from one forward pass into another to causally localize where a behavior lives in a model.

Technical Definition

An interpretability technique that swaps activations from one forward pass into another to causally localize where a behavior lives in a model.

How it works

Also called causal tracing, activation patching runs the model on a clean prompt, caches activations, then runs it on a corrupted prompt while patching in cached activations at chosen sites (residual stream layer L, head H, MLP). The change in output isolates which components causally encode the behavior. It is the core experimental tool of mechanistic interpretability.

Related Concepts

• AI Alignment — The research field aimed at making AI systems pursue the goals their developers and users actually intend.
• Mechanistic Interpretability — A research program that reverse-engineers neural networks into human-understandable circuits, features, and algorithms.
• Sparse Autoencoder (SAE) — An overcomplete autoencoder with an L1 penalty used in interpretability to decompose neural activations into monosemantic features.