WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

WhatWorkedBench. Benchmarking Experimental Understanding in AI Agents. By Jingjie Ning and three others, from Carnegie Mellon University and Tsinghua University. Published on the twenty-third of September. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.

AI research agents need reliable knowledge of how their experiments change outcomes. Running an experiment is one thing; knowing afterwards which change did what is another. WhatWorkedBench measures that second thing, under the name experimental understanding: how accurate an agent's predictions about component changes are, after a budget of experimentation.

The agent inspects code, selects measurements, and submits a response surface, which is a table predicting the score for every configuration of the component settings. The reference comes from exhaustive execution on CPU. It supplies the true effect of changing each component while holding the others fixed, and these effects capture combinations of changes. Because every configuration was actually run, the answer key is complete. An agent's table can be checked cell by cell against what really happens, not against another model's guess.

The benchmark covers thirty-six tasks, drawn from thirty data sources and eight workflow types. Together they hold one thousand two hundred forty-eight configuration records. The core evaluation combines four thousand two hundred six numerical-control records, spanning all eight families. Alongside those sit one hundred eight agent episodes, across the original six families. The numerical-control records give a baseline for how well effects can be inferred from limited measurements, and the agent episodes show how real agents do.

At eight new measurements, a pair-effect ridge method selects an optimum on fifteen of twenty-two sources, and on three of them it keeps every effect error within ten percent of the score range. Fitting a Gaussian process to the same agent observations raises effect recovery, meaning accuracy relative to the true effect magnitude. In the original Flash cohort it goes from zero point six three two to zero point six nine eight. In an additional cohort it goes from zero point six two one to zero point seven two.

On six completed submissions in beat detection and graphs, the same-observation process raises family-macro recovery from zero point three zero three to zero point four five five. On six workflows with six binary options and twenty new measurements, encoding code equivalences, meaning configurations with identical behavior, raises the process's recovery from zero point two four eight to zero point four six two. The pattern across these results is that the same observations can support better or worse conclusions depending on how they are analyzed, and telling the analysis which configurations are really the same helps it.

What the benchmark isolates is whether an agent knows why a result happened, not only that it happened. It is offered as ground for research on experimental agents, adaptive experimental design, numerical inference, and the use of program structure, the last of which the code-equivalence result speaks to directly.