AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

AutoDataBench. A Data-centric Testbed for Accelerating Auto Research. By Ruifeng Yuan and eleven others, from the Hong Kong Polytechnic University, IQuest Research, and six other institutions. Published on the thirtieth of September. Its code repository is worked out: about fifty-five source files, with a readme.

When one frontier agent beats another on an auto-research benchmark, why? The benchmarks that exist often entangle several sources of improvement at once: the training framework, the hyperparameters, the compute budget, and the data. With all of those moving together, a win cannot be attributed to any specific research capability. An agent might come out ahead because it tuned a learning rate well, or because it spent its compute differently, or because it really did improve the data, and the score alone does not say which. Separating those requires a testbed where only one of them is allowed to vary.

AutoDataBench isolates one capability and evaluates it systematically. The authors call it Data Intelligence: an agent's ability to understand, manipulate, and improve the data that shapes what a model can do. Their conceptual framework divides it into data diagnosis, data organization, and data construction.

The testbed instantiates that framework as three highly curated optimization tasks, in tool use, retrieval, and knowledge injection. Every factor other than data is held fixed, which is the whole point: when the framework, hyperparameters and compute cannot move, any improvement has to come from what the agent did to the data. Frontier language models try to improve training data through iterative experimentation, each task with its own resource budget, so they can run an experiment, see what happened, and try again within limits.

Optimization performance is not the only thing measured. The benchmark also asks whether the models understand what their data interventions do. Before training, a model predicts the effect of its change. Those predictions are compared with the observed outcomes, looking for evidence of reasoning about data effects rather than plain trial and error. The authors also explore whether iterative feedback helps the models understand better how changes to training data affect performance.

The abstract does not report the outcome of those comparisons. What it does report is a second use for the benchmark. Trajectories collected on AutoDataBench, reused for mid-training, improve downstream coding performance.

That doubles the testbed's role. It measures data intelligence, and the record of agents exercising it turns out to be high-quality training data in its own right. Underneath both sits the question the benchmark keeps returning to. Not only can an agent make the data better, but does it know why its change worked, or is it guessing until something sticks? Code and resources are on GitHub, under AutoDataBench.