Intermediate · Evaluation

Continuous Evals

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Running evaluation suites continuously in production — on live traffic samples and every model or prompt change — instead of as a one-time pre-launch gate.

Technical Definition

Running evaluation suites continuously in production — on live traffic samples and every model or prompt change — instead of as a one-time pre-launch gate.

How it works

Models drift, prompts get edited, retrieval corpora change, and providers silently update endpoints — so an eval that passed at launch says little about next month. Continuous evals sample production traffic, score it with a mix of model-graded and deterministic checks, alert on regression, and gate every prompt or model change behind the same suite. The practice matured in 2026 into standard CI plumbing: eval results are versioned alongside the prompt and model version, so any regression can be bisected to the change that caused it. The main cost is labeling and judge-model spend, which teams cap by sampling and by reserving human review for disagreements.

Related Concepts

• Outcome-Based Evaluation — Judging AI systems by the quality of real-world outcomes they produce rather than the surface quality of individual answers.
• Evals — Abbreviation for LLM evaluations or any form of evaluation.
• Model Drift — The degradation of a model's performance over time due to changes in the underlying data distribution.