AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents
Dongki Kim, Namkyeong Lee, Surag Nair, Carl Edwards, Xiner Li, Edward De Brouwer, Jenna Lynn Collier, Sung Ju Hwang and 2 others
Genentech · KAIST
Published October 4, 2026 · on the feed October 7, 2026 · ▲ 42 · arXiv 2610.05140
The abstract
As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities. We address this challenge by investigating whether scientific-agent benchmarks can be automatically generated and iteratively adapted as agent capabilities evolve. We introduce AutoSciBench, a framework that represents each task as a high-level concept specifying the scientific domain, data modality, and required reasoning approach, together with a low-level recipe specifying how the question, environment, and ground-truth answer are constructed and verified. Agents attempt to solve each task, producing solver trajectories and corresponding judge feedback which AutoSciBench uses to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories further guides new concept generation, allowing lessons from earlier task refinement to inform subsequent benchmark construction. Starting from existing benchmarks, we evaluate AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science, respectively, while generated tasks receive higher average quality ratings across all three domains, suggesting that scientific-agent evaluation can adapt as agent capabilities advance.
The Brief
2:36A short podcast summary of the paper. Written and voiced by AI.
Transcript
AutoSciBench. Autonomous Benchmark Generation for Evaluating Scientific Agents. By Dongki Kim and nine others, from Genentech and KAIST. Published on the fourth of October. No code repository is linked to it.
Agents are improving fast, and a fixed benchmark can saturate. Once it does, it no longer separates one agent's capabilities from another's, and it stops revealing where agents still fail. Scientific domains make this worse. Building and updating a benchmark there takes a great deal of time, labour and domain expertise, so keeping evaluation in step with what agents can now do is hard. The question the authors pose is whether benchmarks for scientific agents can be generated automatically, and then adapted over and over as the agents change.
AutoSciBench is a framework for producing such benchmarks. Each task is described at two levels. At the high level is a concept, which names the scientific domain, the data modality and the reasoning approach the task requires. At the low level is a recipe, which spells out how the question, the environment and the ground-truth answer are constructed and verified. The concept says what kind of science is being tested. The recipe says how a particular instance of it gets built and checked.
Tasks are then refined against real attempts. Agents try to solve each one, producing solver trajectories and judge feedback. AutoSciBench uses that material to revise the recipe or the concept, closing the shortcuts it observed. The revisions push tasks toward three kinds of work: re-examining raw data, interpreting intermediate results, and integrating evidence.
The process also learns across tasks. Experience distilled from finished refinement trajectories guides the generation of new concepts, so lessons from refining earlier tasks shape how later parts of the benchmark are built. The whole thing starts from existing benchmarks, and is evaluated in three fields: computational biology, materials science and clinical imaging.
The generated benchmarks are harder. Compared with the human-curated benchmarks, average solver accuracy falls by twenty-two point four percentage points in computational biology, and by twenty-five point five percentage points in materials science. They are also rated better. Across all three domains, the generated tasks receive higher average quality ratings.
Those two results need each other. A benchmark can be made harder simply by making it worse, so the drop in accuracy only means something next to the higher quality ratings. Together they are the basis for the authors' suggestion that evaluation of scientific agents can adapt as the agents themselves advance, rather than waiting for people to rebuild it.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Benchmarks age fast when agents improve faster. Here is the wall.
You evaluate AI agents that analyse scientific data. Agents improve fast, and the benchmark you rely on is saturating: top agents score near the ceiling, so it no longer separates them or shows where they still fail.
The road ahead
- 1 · The problem ~7 min Scientific-agent benchmarks saturate as agents improve, and writing new expert tasks by hand is too slow to keep evaluation in step.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Describe each task as a concept plus a recipe, refine it by watching agents attempt it, and carry distilled lessons into new concepts.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Pick a concept, plan a recipe, build and test the task with solvers and a judge, revise to close shortcuts, then distill experience for new concepts.
- 6 · The experiment ~6 min You’ll guess: Compared with the human-curated benchmark in materials science, by how many percentage points did generated tasks lower average solver accuracy?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.