TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces

TraceDance. An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces. By Dehai Min and fifteen others, from ByteDance and the University of Illinois at Chicago. Published on the twenty-seventh of September. Its code repository is public but slight: three source files.

An agent can finish its task and still behave badly on the way there. Success at the end hides what happened in the middle. Developers who see a particular undesirable behaviour in deployment need tests for that behaviour, and a fixed benchmark suite, written before the behaviour showed up, will not necessarily contain one.

TraceDance is an agent system that builds those tests on request. A user names an undesirable behaviour, and the system constructs a targeted benchmark for it out of real deployment traces. What each benchmark then measures is narrow and concrete: given a recorded decision point, does a model's next turn respond appropriately, judged against a rubric written for that specific behaviour?

Construction has to be efficient, and two components handle it. Anchor-and-Confirm pairs programmable retrieval with confirmation at the level of individual candidates, carried out by a Flash large language model. For behaviours that need custom handling, the Anchor Synthesis Loop generates specifications and revises them. Evaluation uses what the authors call decision-point continuation. The model under test picks up at a recorded decision point and produces its next turn, which is graded with the behaviour-specific rubric. There is no reference answer, and the environment is never replayed.

The experiments cover coding and general tool use. From two hundred fifty-two thousand five hundred fifty-seven sessions, the system produces one hundred and seven benchmarks containing four thousand one hundred twenty-five instances, and fulfils ninety-five point three percent of the requests for a build target. Quality is checked by people. Both human annotators confirm the requested behaviour in eighty-four percent of sampled instances, and the automated grader agrees with human pass or fail judgments about as well as the two annotators agree with each other.

Nine frontier LLMs are then evaluated, and their mean pass rate is only twenty-six point seven percent. At these decision points, they still struggle to respond appropriately. Analysis across the behaviour-specific benchmarks exposes further weaknesses in how current LLMs act as agents.

The pipeline runs from a problem seen in deployment, to a benchmark, to a score, with no one hand-writing the test in between. That is why the authors place it inside a recursive self-improvement loop, RSI in their shorthand: each deployment problem can become a targeted measurement that the next round of improvement is judged against.