FrontierPhysics: Evaluating Agents for End-to-End Physics Research
Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen, Min Chen, Shuqi Xu, Yue Ma and 64 others
BenchFlow · University of California, Berkeley · California Institute of Technology · Harvard University · University of Washington · Stanford University
Published November 4, 2026 · on the feed November 5, 2026 · ▲ 59 · arXiv 2611.06910
The abstract
AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.
The Brief
1:40A short podcast summary of the paper. Written and voiced by AI.
Transcript
FrontierPhysics. Evaluating Agents for End-to-End Physics Research. By Bingran You and seventy-one others, from BenchFlow, the University of California, Berkeley, and thirty-five other institutions. Published on the fourth of November.
AI agents are playing an increasingly important role in physics research. The benchmarks used to judge them have not kept pace. Existing physics benchmarks evaluate fixed questions, isolated sub-tasks, or reproduction of a paper whose solution is already given. And they rarely check the quality of the research alongside whether it is correct.
FrontierPhysics evaluates research end to end. Each task requires an agent to review the literature, develop a research plan, conduct the research and experiments, and produce structured outputs along with a scientific manuscript. What it asks of the result is twofold: is the research correct, and is it publishable?
There are fifty-six tasks, made by forty-two physicists from their own research. They cover nine domains across theoretical, computational and experimental physics. Grading combines deterministic verifiers with rubrics authored by experts.
Twenty-two models were evaluated, in twenty-six configurations. No configuration passes more than forty-six of its one hundred and sixty-eight rollouts. Manual review of the agents' trajectories then locates the weakness. It is not a gap in physics knowledge, and it is not errors of execution. Frontier agents fall short mainly on reasoning that is insufficiently rigorous, and on expert intuition that is limited.
Read together, the design and the failures line up. A benchmark built from working physicists' own problems, and graded for publishability as well as correctness, finds agents whose trouble is not knowing the physics or carrying out the steps, but reasoning with the rigour and intuition an expert brings.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Answering physics questions is not doing physics. How would you test the difference?
You want to know whether AI agents can really do physics research. The tests on offer ask fixed questions, check isolated sub-tasks, or ask an agent to reproduce a paper whose solution is already known.
The road ahead
- 1 · The problem ~7 min Existing physics tests for AI agents check fixed questions, small pieces, or guided reproductions, and rarely ask whether the research itself is any good.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give agents whole research projects written by physicists from their own work, then grade both correctness and whether the research is publishable.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min An agent reviews literature, plans, does the research, delivers outputs and a paper, and is then graded by verifiers and a rubric-applying reviewer agent.
- 6 · The experiment ~6 min You’ll guess: Manual review of frontier agents' research attempts found one main kind of weakness. Which?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.