Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

Automating ML for Science. Can Frontier Agents Climb Scientific Hills in the Wild?. By Ming Zhong and three others, from the Anthropic Fellows Program and Anthropic. Published on the twentieth of October. Its code repository is public and real: about twenty source files.

Frontier coding agents are increasingly described as end-to-end automated researchers. Machine learning for science is a demanding place to test that description, because improving a scientific model often takes domain insight that goes beyond routine machine-learning engineering. An agent can be a capable engineer and still have nothing new to say about the science.

Agent4Sci is a long-horizon testbed that measures how far today's agents can improve on the published best models for scientific machine-learning benchmarks. It is deliberately not a leaderboard. Rather than ranking agents, it asks what kinds of progress agents make on these tasks under in-the-wild conditions, and where they fall short. Progress is reported as relative improvement over the published best, and runs are checked for academic misconduct.

The testbed spans twenty benchmarks across eleven domains. Each one starts from a published best model, and the agent's job is to push past it over a long horizon, with its trajectory recorded so that how it got there can be analysed as well as how far.

Today's frontier agents do advance the published best on every benchmark. Claude Opus reaches a mean relative improvement of ten point four percent, and no run is flagged for academic misconduct. But the gains come almost entirely from a bag of machine-learning engineering tricks rather than from research-level insight. Trajectory analysis surfaces three recurring limitations: novelty miscalibration, under-use of available resources, and a rapid collapse into local engineering sweeps. Put together, they describe an agent that misjudges how new its own ideas are, leaves resources on the table, and soon settles into tuning around wherever it already is, instead of stepping back to try something different.

Lightweight plug-in skills help. Adding skills for research ideation, experiment management and scheduled checklists raises Claude Opus from ten point four percent to twenty point one. Codex rises from four point four percent to eight point one. The interventions also become more diverse and more relevant to the domain. For Claude Opus the mean improvement nearly doubles, and for Codex it does much the same.

The answer to the title's question is yes, in a particular way. Agents do climb the hills, but mostly by engineering rather than by insight. The testbed is offered as a way for the community to track that balance, and to push frontier agents from machine-learning engineering toward domain insight on real problems in machine learning for science.