AI teams have inherited a seductive idea from machine learning research: if a model’s benchmark score rises, the product has improved. That assumption is convenient, comparable and frequently wrong. A benchmark measures performance under a particular protocol. A product succeeds inside a changing environment populated by ambiguous requests, partial data, brittle integrations and users who behave nothing like test-set authors.
The distinction becomes critical as language models graduate from answering questions to operating tools. A model may ace a reasoning test and still select the wrong customer record, retry a payment twice or confidently summarize an outdated document. These are not exotic edge cases. They are the ordinary failure modes of software assembled from probabilistic components.
The research community already knows that evaluation is harder than producing a single accuracy figure. Anthropic’s account of the challenges in evaluating AI systems describes how formatting choices, implementation differences and flawed questions can materially affect results. The lesson for product teams is sharper: if a supposedly standardized academic test is sensitive to its plumbing, an evaluation built around a company’s shifting workflows will be even more sensitive.
An evaluation should diagnose, not decorate
Most internal evaluation suites begin as release gates. A team collects a few dozen examples, defines an aggregate pass rate and prevents deployment when the number falls. That is better than intuition, but it soon turns into dashboard theater. The score remains stable while the application changes underneath it. New tools appear, prompts accumulate exceptions and users invent workflows the original examples never represented.
A useful evaluation program resembles an instrument panel rather than a school exam. It separates retrieval errors from reasoning errors, policy violations from tool failures, and recoverable mistakes from irreversible ones. It records latency and cost alongside answer quality. Most importantly, it retains the trace: the context supplied, tools called, intermediate decisions made and state changed. A failing output without its execution history is a bug report missing the stack trace.
Consider an assistant that prepares a sales proposal. “Did it produce a good proposal?” is an attractive but weak test. A diagnostic suite asks whether it selected the current price list, distinguished approved claims from internal notes, preserved required contract language, cited the correct customer facts and requested approval before applying a discount. Those checks identify where to intervene. The fix may be a better model, but it may instead be stricter retrieval, a typed tool interface or one additional authorization boundary.
This is also why realistic evaluations cannot consist only of questions with short answers. The GAIA benchmark was notable for testing assistants on conceptually simple tasks that combine reasoning, browsing and tool use. Its deeper contribution was architectural: once success depends on interacting with an environment, evaluation must cover the whole system rather than the model in isolation.
Build an error portfolio
XioX’s view is that every serious AI product needs an error portfolio: a maintained collection of representative failures weighted by business consequence. The portfolio should include frequent annoyances, rare dangerous cases and examples that once caused regressions. It should grow from support conversations, production traces, adversarial testing and domain-expert review—not merely from synthetic prompts generated before launch.
The word “portfolio” matters. Teams should not optimize every failure equally. A slightly awkward rewrite and an unauthorized database update do not belong in the same average. Segment tests by severity, reversibility and exposure. Require near-perfect handling where an action is costly or difficult to undo, while accepting graceful imperfection in low-stakes drafting. An honest system communicates uncertainty and stops safely; an over-optimized demo attempts everything.
Evaluation data also has a half-life. Once a suite guides model selection and prompt changes, teams begin adapting to it. The suite gradually becomes a training target, even without formal fine-tuning. Hold back cases, rotate surface details, add newly observed distributions and periodically ask specialists to attack the system from first principles. Passing yesterday’s regression set proves only that yesterday’s failures stayed fixed.
The organizational test
The hardest question is not which framework to install. It is who owns the definition of acceptable behavior. Engineers can measure deterministic properties, but product operators understand workflow reality, security teams understand abuse paths and domain experts know which plausible answer is dangerously wrong. Evaluation is therefore a governance function expressed through code.
Teams that treat it as a final quality-assurance phase will always lag behind their product. Teams that treat it as a living specification gain something more valuable than a leaderboard win: a shared, executable account of what the system is allowed to do, how well it must do it and what happens when it cannot. The benchmark is evidence. The instrument is the capability.
Advertisement
