From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research

From Research Goals to Scientific Closure. Evaluating Agents Through Evidence-Supported Physics Research. By Yijia Mao and sixteen others, from the Shanghai AI Laboratory, Shanghai Jiao Tong University, and two other institutions. Published on the fifteenth of October. Its code repository is public and real: thirteen source files.

The question is whether AI agents can organise a scientific investigation that ends in a well-supported conclusion. Doing the pieces of research well is not the same thing. An agent can carry out a sound step, or produce a result that looks right, and still leave the actual question unresolved. Scoring only the pieces would miss that difference, and the difference is what this benchmark is built to expose.

PhysClosureBench measures on two levels. A twenty-criterion Atomic Score captures local scientific accomplishment, reported out of ten. Research Closure, graded from RC0 to RC4, asks something else: how fully the investigation resolves the research question it was given. That grade depends on whether the investigation satisfied the question's indispensable evidence requirements, and its applicable scope conditions. RC4 is the top of the scale. Pairing the two lets the benchmark tell apart an attempt that does many things well from one that actually finishes the job.

The benchmark holds fifty research tasks, each verified by experts and each specified by its goal. They span strong-field physics, attosecond physics and neighbouring areas. A task states the research objective and the essential scientific constraints. Everything else is left to the agent: how to organise the investigation, and how to generate and validate the evidence needed to answer the question.

Seven frontier LLMs are evaluated. Claude-Opus-5 has the highest mean Atomic Score, six point two nine out of ten, yet reaches RC4 in only thirty-two point seven percent of its attempts. Across the whole benchmark, thirty-six point two percent of attempts scoring at least eight out of ten still fall short of RC4.

Consistency is a second weakness. Every one of the seven models reaches RC4 on substantially fewer tasks in all three of its runs than in at least one of them. Case-level analysis points to two ways closure breaks down: agents misinterpret results they generated themselves, and they leave limits on physical applicability unresolved. Either one can leave an investigation incomplete.

What the numbers describe is a persistent gap. Agents accomplish a good deal locally, criterion by criterion, and still struggle to bring an investigation to complete closure. On this evidence, a high Atomic Score is not the same thing as an answered question.