Intermediate · Evaluation

Outcome-Based Evaluation

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Judging AI systems by the quality of real-world outcomes they produce rather than the surface quality of individual answers.

Technical Definition

Judging AI systems by the quality of real-world outcomes they produce rather than the surface quality of individual answers.

How it works

As agents took on multi-day tasks, per-response metrics (helpfulness ratings, single-turn benchmarks) stopped predicting value. Outcome-based evaluation measures the end result: did the code pass CI in production, did the research report change a decision, did the support workflow resolve the ticket without escalation. It requires longer evaluation horizons, environment-level instrumentation, and causal care — an agent can produce a lucky good outcome or an unlucky good process. Mature setups pair outcome metrics with trajectory audits so teams can tell skill from luck.

Related Concepts

• Persistent Background Agents — Long-running autonomous agents — such as Meta's Muse Code and Replit's Agent 4 — that build, debug, and ship software over days from a single natural-language instruction.
• Evals — Abbreviation for LLM evaluations or any form of evaluation.