Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Trial by Trace. Benchmarking AI Agent Harnesses for Reliability and Failure Attribution. By Zhiyong Cao and twenty-two others, from the Institute of Automation, Chinese Academy of Sciences, Zhongguancun Academy, and three other institutions. Published on the twenty-third of October. Its code repository is worked out: about four hundred source files, with a readme and a licence.

LLM agents are increasingly used to solve real-world problems, and how well they do depends on two things: the underlying model, and the harness that supports its operation. Most existing benchmarks focus on model capabilities and on overall agent performance. The harness itself, the layer that runs the model, gets limited attention. When an agent fails, there is little to say whether the model or its harness was at fault. And when an agent succeeds, there is little to say whether the harness did its job along the way, or the run simply got lucky. The authors set out to close that gap with a benchmark aimed at the harness.

TraceTrial-Bench organises harness reliability into seven components: context, execution, governance, lifecycle, observability, tools and verification. Its two-layer evaluation framework assesses agent performance and harness reliability separately. A run can therefore be judged on whether the task got done, and independently on whether the harness behaved properly while doing it. Splitting the two is the core of the design. A single pass or fail number would merge them back together, and the harness would disappear behind the model again.

There are ninety-five expert-level, mechanism-oriented tasks, each with executable or state-based ground truth. That supports semantically equivalent comparisons across harnesses, while letting each one keep its native behavior. Three modules carry the evaluation. A Trajectory Tracing module captures agent execution traces. A Multi-source Evidence Acquisition module supplies evidence from the agent context, the harness and the execution environment to an LLM-based evaluator. And a Failure Attribution module uses explicit mechanism chains to support attribution at the level of the individual run.

The evaluation covers four harnesses, nine models and ninety-five tasks, for three thousand four hundred and twenty runs in all. Macro-averaged harness scores range from fifty-nine point eight percent to seventy-two percent. Alongside that, thirty-one point one percent of runs achieve task success despite failing at least one harness criterion. Harnesses differ, then, even when the models and tasks are held in common, and task success on its own is a poor guide to which runs were sound.

That last figure is why the second layer exists. Nearly a third of all runs reach the goal while breaking a harness criterion on the way, and a score that looked only at outcomes would count every one of them as clean. The authors present the benchmark as an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.