The most dangerous number in an AI project is often the one that looks most scientific: an aggregate benchmark score. It arrives with decimal places, a clean comparison table, and the reassuring air of an objective verdict. Model A scored 84; Model B scored 81; procurement can proceed.
That is usually where clear thinking stops.
A benchmark is not a universal measure of intelligence. It is a sensor built to detect certain behaviors under certain conditions. Like any sensor, it has a range, a resolution, and blind spots. The useful question is not which model sits three points higher. It is whether the instrument measures the failure modes that matter in the system you intend to operate.
The distinction matters because production AI is conditional. A model may answer isolated questions well yet become unreliable when evidence is scattered across a long document. It may write plausible code but mishandle an unfamiliar repository. It may select the correct tool in a clean test and call the wrong tool after twenty noisy interactions. None of those differences can be captured honestly by one blended score.
Evaluation begins with a theory of failure
The influential HELM framework helped establish a broader view of evaluation by considering multiple scenarios and dimensions such as accuracy, calibration, robustness, fairness, and efficiency. Its durable lesson is not that every team needs a giant benchmark suite. It is that quality has multiple axes, and compressing them too early conceals trade-offs.
Product teams should begin by writing a failure taxonomy. For a support assistant, that might include unsupported refunds, policy hallucinations, privacy leaks, needless escalation, and technically correct answers delivered too late. For a coding agent, it might include regressions outside the edited module, unnecessary dependency changes, commands executed in the wrong directory, and tests that pass only because they fail to run.
These are not merely test cases. They are hypotheses about how the system can disappoint a user or damage an environment. Each hypothesis should determine the dataset, grader, and operating condition used to test it. If a failure would be expensive, the evaluation deserves more repetitions and stricter evidence.
This approach also exposes a common mistake: testing the model while ignoring the system. Prompt templates, retrieval, tool descriptions, context management, retry logic, and permission rules all influence the outcome. Anthropic’s practical discussion of agent evaluations emphasizes that multi-step systems modify state and adapt to intermediate results. Judging only the final sentence throws away the evidence needed to understand what happened.
Measure trajectories, not just answers
An agent can reach a correct result through a reckless process. It might search an unauthorized source, expose sensitive context to an unnecessary tool, make several harmful edits, and then repair them before the grader inspects the final state. A result-only test calls that success. An operational evaluation should not.
Useful agent telemetry includes tool calls, arguments, state transitions, retries, reversals, latency, cost, and the evidence consulted before a consequential action. The trajectory reveals whether the system is competent or merely lucky. It also makes failures diagnosable: a retrieval problem demands a different intervention from a reasoning problem, even when both produce the same wrong answer.
Long-horizon performance introduces another complication. Reliability compounds. A system that performs each step well can still struggle when success requires many dependent decisions. METR’s work on measuring task-completion horizons offers a more operational framing than trivia-style accuracy: ask how long a task can be before dependable completion collapses. Whether or not that exact metric fits a product, the underlying idea is valuable. Difficulty is not only about intellectual complexity; it is also about duration, branching, and opportunities for error.
A good eval changes an engineering decision
Evaluation programs often become museums: carefully maintained collections of scores that nobody uses to block a release, select an architecture, or revise a workflow. That is measurement theater.
Every evaluation should have an owner and a decision attached to it. A threshold might determine whether a model can send an email without review, whether retrieval needs another source, or whether a smaller model can replace a costly one for a narrow task. Tests without decision rules may be interesting research, but they are weak production controls.
The best suite contains three layers. Stable regression tests protect known requirements. Adversarial tests probe anticipated failures and should evolve as the system changes. Production sampling examines the messy distribution that designers failed to imagine. These layers should disagree occasionally. If they never do, the tests are probably too similar.
Teams should also preserve slices instead of celebrating averages. A five-point improvement among routine requests can hide a severe decline on rare, high-impact cases. Report performance by task type, language, context length, tool, risk tier, and ambiguity. Aggregate only after examining the shape underneath.
At XioX, we think the mature evaluation question is not “How smart is this model?” It is “What evidence would justify trusting this system with this action, under these conditions?” That question is narrower, harder, and far more useful. A scoreboard produces a winner. A sensor tells engineers when the machine is unsafe to run.
Advertisement
