A coding agent can fail while producing a perfectly plausible patch. It can misunderstand the issue, edit the wrong abstraction, satisfy a narrow test, and leave the repository in a state that costs a senior engineer an afternoon to untangle. Yet much of the evaluation culture around agents still compresses this messy trajectory into a binary result: resolved or unresolved.
That made sense when the research question was whether language models could modify real repositories at all. The original SWE-bench paper was a major advance because it moved evaluation beyond isolated code snippets and into real issues, codebases, tests, and patches. But once agents enter everyday development, passing the final test suite is no longer a sufficient definition of success. The path matters.
Software engineering is full of partial information. Requirements contradict implementation details. Tests encode historical accidents. A clean-looking fix can violate an undocumented contract several directories away. Human engineers cope by forming hypotheses, testing them cheaply, revising their mental model, and escalating when uncertainty remains. An agent that cannot recover from an early false belief is not autonomous; it is merely persistent.
Reliability compounds across the trajectory
Long tasks expose a simple mathematical problem. If an agent has a high probability of choosing correctly at each consequential step, its chance of completing a long sequence without intervention can still fall quickly as those steps accumulate. Real trajectories are not independent coin flips, but the intuition holds: modest local error rates become expensive over dozens of tool calls, edits, and interpretations.
Research from METR on task-completion time horizons offers a better lens than one-shot accuracy. It asks how task length relates to reliable completion, recognizing that agents often possess the component skills but struggle to string them together. That distinction should reshape coding evaluations. We need to know not only whether an agent can implement a fix, but how its behavior changes after the first failed test, misleading search result, or rejected hypothesis.
Consider two agents that both solve seven of ten issues. Agent A reaches successful patches quickly when its first theory is right, but contaminates the branch with scattered edits when it is wrong. Agent B spends longer inspecting the repository, keeps changes isolated, and reliably returns to a clean state after failed experiments. A leaderboard records a tie. An engineering manager should strongly prefer Agent B.
Recovery is observable
Recovery sounds subjective, but it can be measured through concrete events. Did the agent detect that a test failure contradicted its current theory? Did it revert speculative changes that were no longer justified? Did it preserve unrelated work? Did it repeat a failed action without gaining information? Did it broaden the scope of edits before establishing a causal link? Did it ask for help while the state was still legible?
A useful evaluation suite would deliberately introduce recoverable disruptions. A test could fail for a reason unrelated to the patch. An issue description could contain a plausible but incorrect diagnosis. A dependency could expose two similar APIs, only one of which is appropriate for the repository’s version. The goal would not be to trick the agent. It would be to reproduce the ambiguity that defines maintenance work.
Fresh tasks matter too. Static benchmarks can leak into training data, documentation, and public traces. The SWE-bench-Live proposal responds by drawing from newer repository activity. Continuously refreshed tasks are valuable, but freshness alone does not capture operational safety. A new issue can still be scored with an old idea of success.
Give every run a recovery ledger
We propose attaching a recovery ledger to each agent run. Alongside the final outcome, it would record state-changing actions, failed hypotheses, rollback quality, repeated errors, test selection, and the point at which human assistance became necessary. Evaluators could then distinguish clean failure from destructive failure and robust success from lucky success.
The ledger should include a cleanup cost: how much work must a competent engineer perform before continuing from the agent’s final state? This is not the same as reviewing the patch. Sometimes the patch is wrong but neatly contained, making recovery trivial. Sometimes it passes while leaving unexplained generated files, weakened assertions, or broad formatting churn. Those differences determine whether an agent accelerates a team or creates invisible inventory.
This approach would also reward better agent architecture. Checkpointing, explicit uncertainty tracking, reversible operations, and hypothesis-aware test selection may not improve the headline pass rate immediately. They can nevertheless make systems far more deployable. Today’s benchmarks often undervalue these features because they treat the repository at the end of the run as the only artifact that matters.
The next phase of coding-agent research should stop asking only, “Did it fix the issue?” The more revealing question is, “When it was wrong, did it leave the work in a condition from which intelligence—human or machine—could continue?” Production software is not a sequence of pristine puzzles. It is an ongoing negotiation with incomplete knowledge. The agents worth trusting will be the ones that know how to retreat.
Advertisement
