SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

SciΠvot. Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents. By Youran Sun and seventeen others, from the University of Maryland, College Park, the Chinese University of Hong Kong, and four other institutions. Published on the fifteenth of October. No code repository is linked to it.

Language model agents can now keep scientific experiments running for months with little human supervision. Keeping experiments running is not the same as doing research well, though. Reliable autonomous research also needs scientific judgment, and judgment is hard to evaluate, because research decisions are open-ended and depend on the context of a particular project. There is rarely a clean answer key.

SciΠvot gets its answer key from people. It measures scientific judgment against human corrections: moments when a senior researcher looked at what an agent planned to do and overruled it. Each such moment becomes a test of whether a model, standing where the agent stood, would pick the plan the human preferred. Alongside the raw score, it measures how that choice shifts with reasoning effort, with access to the project's history, and with a chance to reconsider, and it sets model accuracy beside that of human researchers.

The records come from six months of research, in which senior researchers ran language model agents on twenty-six projects across six scientific fields and repeatedly stepped in to correct the agents' research judgments. From those interventions the authors build two hundred and forty-eight cases. In each case, the model being tested chooses between two options, the original LLM plan and the human-corrected alternative, and it sees only the project record that existed before the correction. Nothing from after the intervention leaks into the context. The setup is a binary choice, so chance performance sits at half, and that gives every score below a fixed point of reference.

At medium reasoning effort with the project record in hand, models systematically prefer the original LLM plan over the human correction. Four of five frontier models score below chance on the selected cases. The best accuracy any tested model reaches in the project condition, across every reasoning effort setting, is sixty-three point five percent. Fresh researchers reach seventy-three point four percent.

Context and second chances do not rescue them. At medium effort, every model is less accurate with the project record than without it, so more context made the choice worse rather than better. And giving a model the chance to reconsider its answer does not consistently raise overall accuracy either.

The task is deliberately generous: the right answer is sitting on the table as one of two options. Even so, frontier models struggle with these research decisions, and lean toward the plan an LLM originally made. SciΠvot gives a way to evaluate that kind of judgment systematically, drawn from the moments when human researchers actually had to step in.