OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

OpenFrontiers. A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery. By Jiarui Yao and two others, from the University of Illinois Urbana-Champaign. Published on the twenty-seventh of October.

Large language models show growing promise at research-level problem solving, and yet two things are missing. There is no comprehensive collection of such problems to point them at. And evaluating a model on a research-level problem remains challenging in its own right. A research problem does not come with an answer key in the way an exam question does, and once a problem has been solved and published, a model that reaches the answer may have found it, or may simply have looked it up.

OpenFrontiers answers both with one cross-disciplinary resource in two parts. The first is a structured corpus of open research problems, recording the problem statement, its context, progress made so far and other information, for twelve thousand one hundred and fifty-one problems. The second is a retrospective benchmark built from mathematics problems that were recently resolved. What it measures is rediscovery: can an agent reach a known answer that appeared after the problem was posed.

The benchmark holds two hundred problems. A hundred and ten were resolved in two thousand twenty-six. The other ninety remain open. Each carries a Lean four statement, checked by automated probes and by two independent reviews, and a subset called HQ-50 filters further on a quality score. Grading is formal. Proofs are checked by two independent checkers inside an isolated sandbox, under three conditions: closed-book, frozen retrieval, and live web access. Those three conditions are the core of the design. They separate what an agent can do from its own knowledge, what it can do with a fixed snapshot to search, and what it can do with the open web, where published solutions may be waiting.

Seven large language model agents were evaluated on HQ-50 under a standardized budget. Closed-book rediscovery turns out to be rare: at most seven of the forty resolved problems. With live web access, agents succeed more often, but that success mostly reflects retrieving solutions that have already been published, not deriving them. The ninety open problems sit alongside as the other half of the benchmark, problems with no known answer to retrieve at all.

The contrast between those two conditions is the clearest result here. Give an agent the web and it looks capable; take the web away and very little of that capability is its own. By tying a broad corpus of research problems to a formally graded subset with documented solutions, OpenFrontiers offers a footing for measuring rediscovery, and for tracking progress toward problems that are still open.