SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

SciGame. Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World. By Yusheng Zhao and three others, from MBZUAI, McGill University, and one other institution. Published on the twenty-seventh of October.

Large language model agents are increasingly being used for scientific discovery, and the benchmarks that test them have three problems. They mix inferring a law together with programming. They score how well an agent fits the data, instead of how well it predicts on inputs it has not seen. And they do not separate what an agent learned from empirical evidence from what it already carried in from its pretrained priors.

SciGame is a controlled benchmark built to pull those apart. It converts scientific literature into structured scientific cards. Each card yields a pair of environments with the same task interface: an Original one, and a Counterfactual one that remains dimensionally consistent. Because the two look alike from the outside, an agent that answers from memory instead of from its experiments has nowhere to hide once the law underneath has changed.

The benchmark has two collections. A Historical collection holds canonical laws. A live Temporal collection is curated from papers published after model knowledge cutoffs. Agents are evaluated in two settings, candidate-guided Identification and open-ended Discovery, and each setting runs under three protocols, called Prior-only, Fixed, and Adaptive.

Experiments on five frontier models yield three findings. The first is that familiar priors hurt performance once the governing relation is altered. The second is that adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. The third is that good explanations can hide quantitative failures. Running through all three, recovering the exact symbolic form remains the main bottleneck, even when agents interpret the evidence correctly.

Current agents, then, can investigate a system actively. They run experiments, read the results, and explain them plausibly. What they cannot yet reliably do is the last step, inducing the exact law, and that gap is widest in an unfamiliar setting, where what they remember from training points the wrong way. The authors leave it as an open challenge.