AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?

AutoDataBench. Can Agents Write the Data That Feeds the Self-Improvement Loop?. By Haotian Luo and seven others, from the Hong Kong University of Science and Technology, SLAI, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about seven hundred source files, with a readme and a licence.

Recent gains in language model capability have come more from data than from architecture. Frontier labs and data companies produce verifiable agentic tasks, and supervised finetuning and reinforcement learning turn those tasks into capability. That production line still depends on human labour and on humans in the loop. If agents could create the tasks themselves, data production would scale with compute rather than with the number of experts, it would reach more domains, and it would enable a key step in recursive self-improvement.

The way this ability is measured today does not fit how data is actually bought. Existing evaluations train a model on what an agent produced and then measure that model. In the data industry, data arrives sample by sample, and each sample is accepted or rejected against a set of criteria before it ever goes near training. No existing evaluation asks the industry's question: does this individual task meet the acceptance criteria of a data pipeline? Training on everything an agent delivered and checking the result says something about the batch, not about any single item in it. A buyer at the gate needs a verdict on each item as it arrives.

AutoDataBench asks exactly that. It measures whether an agent can write a single new task that would pass practical acceptance standards. Those standards cover four things: validity, novelty, difficulty, and behavioural coverage. Put plainly, the task has to work, it has to be new, it has to be hard enough to matter, and it has to exercise a broad enough range of behaviour. Scores are reported out of one hundred.

The setup gives the agent an original benchmark task, together with a record of the target model attempting it. From that, the agent must write a new task for the same suite. The suites come from three benchmarks of executable agent tasks. Each candidate is judged on its own, one artifact at a time, against the criteria a production pipeline would apply, and no training run is needed to score it. The default time budget is forty-five minutes.

At that default budget, no agent evaluated scores above twenty out of one hundred. Giving the strongest agent four times as long improves its score substantially. The cost of one usable task, however, stays almost unchanged. More time buys more good tasks, roughly in proportion, rather than making each one cheaper. The authors' reading is that current agents can write training tasks of the required quality, but not efficiently.

Two numbers carry the result. A low score at the default budget, and a cost per usable task that hardly moves when time is quadrupled. Together they say the bottleneck is not whether an agent can ever produce an acceptable task, but how much it spends for each one. AutoDataBench offers a direct measure of autonomous data synthesis that sits where the data industry already makes its decisions: at the gate, sample by sample.