AI teams have inherited a comforting habit from conventional machine learning: choose a benchmark, improve the score, and call the new system better. That habit becomes dangerous when the model is an agent. A classifier returns an answer. An agent navigates an environment, selects tools, modifies state, reacts to failures, and decides when it has done enough. Its behavior is partly produced by the model and partly by everything surrounding it.
This is why a leaderboard result is increasingly a property of a whole test apparatus, not a portable fact about a model. The prompt, tool descriptions, retry policy, available compute, repository setup, grader, and stopping condition can all change the outcome. Put the same model inside another harness and you may discover that the impressive capability was conditional on choices hidden below the headline.
The original SWE-bench paper made an important move away from toy code generation by asking models to resolve real issues in real repositories. That realism matters. But even a realistic benchmark remains a compressed simulation of software work. Production engineering includes ambiguous intent, undocumented conventions, intermittent dependencies, risky migrations, conflicting tests, and decisions whose correctness cannot be reduced to whether a patch passes today.
An agent has a larger surface area than its answer
For an agent, the final artifact is only one layer of quality. The path matters too. Did it inspect the relevant files before editing? Did it notice that a command was destructive? Did it repeatedly call an expensive tool after receiving equivalent failures? Did it preserve user changes? Did it stop because the work was complete, because it exhausted its context, or because it mistakenly believed a partial result was success?
A useful evaluation therefore needs several kinds of evidence. Outcome checks determine whether the task was completed. Trajectory checks inspect how the agent acted. State checks confirm that the environment was left in an acceptable condition. Economic checks measure latency, token use, and tool cost. Human checks address qualities that executable graders cannot fully capture, such as whether a code change respects the architecture rather than merely satisfying visible tests.
This broader view follows the spirit of HELM’s multi-metric evaluation framework: performance is not one scalar, and comparisons become misleading when scenarios and metrics are selectively reported. Agents make that principle more urgent because they can achieve the same outcome through radically different—and differently risky—routes.
Consider an agent asked to update a billing service. Two runs may both produce passing tests. One makes a narrow change after tracing the existing calculation. The other rewrites a shared utility, silently alters rounding behavior, and adds a brittle test that blesses its own mistake. A binary pass rate treats these runs as equivalent. A product team cannot.
Turn product requirements into executable cases
The practical answer is not to build a grand universal benchmark. It is to treat evaluations as executable product specifications. Every important promise made to a user should generate cases that can be replayed: the agent asks before sending an external message; it never modifies files outside an allowed directory; it reports uncertainty when records conflict; it can recover after a tool timeout; it does not declare success when a deployment has failed.
These cases should be drawn from three places. First, happy-path workflows establish that the product can perform its central job. Second, boundary cases exercise permissions, incomplete context, malformed inputs, and irreversible actions. Third, production failures become permanent regression cases. That last category is especially valuable. An incident should not end with a prompt tweak and a relieved Slack message; it should leave behind a durable test.
Anthropic’s guidance on agent evaluations emphasizes combining graders because open-ended trajectories rarely admit a single reliable judge. We agree, but would push the organizational implication further: eval ownership cannot remain confined to a research team. Product managers must define acceptable outcomes, domain experts must identify subtle failure modes, engineers must make environments reproducible, and operators must feed production evidence back into the suite.
The strongest moat may be a private distribution of reality
Models will continue to change, and public benchmarks will continue to saturate. A company’s durable advantage is more likely to be its carefully maintained distribution of real tasks, constraints, and failure histories. That evaluation corpus encodes what the organization has learned about its customers and operations. It lets teams compare a larger model, a smaller model, a new tool interface, or a different workflow against the same grounded definition of usefulness.
This changes procurement too. Instead of asking which model is “best,” teams can ask which system configuration performs best on their work at an acceptable cost and risk. Sometimes the answer will be a frontier model. Sometimes it will be a cheaper model inside a more constrained workflow. Sometimes the evaluation will show that autonomy adds no value at all.
The benchmark still has a role: it reveals general progress and supplies shared scientific reference points. It just cannot carry the product decision by itself. The closer an AI system gets to taking action, the more evaluation must resemble software assurance rather than a school exam. The real question is no longer whether the agent can pass a test. It is whether the test expresses the world in which the agent is expected to earn trust.
Advertisement
