ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

ScholarCatalyst. A Benchmark for Retrieving Papers That Inspire New Research. By Sohyeon Kim and thirteen others, from Stanford University, Seoul National University, and four other institutions. Published on the first of October. Its code repository is public and real: about thirty source files.

What makes great scientists great? The authors point to one skill in particular. Even as AI systems begin to make progress on open problems, scientists stay far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem actually needs. Studying that skill requires knowing, for a real project, which earlier work truly helped, and the people who know that firsthand are the researchers who did the work. An outsider can guess which earlier papers look relevant. The author knows which ones actually moved the project forward.

ScholarCatalyst turns that knowledge into a retrieval task. A system is given the initial research question of a project, and has to retrieve the papers that advanced it. It may search only the literature that existed when the project began. Papers serve as pointers to the ideas inside them, and the judgments of what counts come from the authors themselves. The time limit is what makes it a test of foresight. The system has to find the ingredients as they stood on the day the project started, without the finished project to point the way.

An automated pipeline makes author annotation scalable. Using it, one hundred and eighty-four lead authors of two hundred and seven recent computer science papers labelled which candidate papers did advance their project, or could have, and each label carries a detailed rationale. Including papers that could have helped, not only those that did, means the answer key is not limited to what the authors happened to find at the time.

Agentic search does no better than plain embedding retrieval. Its recall at twenty is zero point four two, against zero point four eight for embeddings, even though the agent calls that very same retriever as one of its tools. An agent built on Claude Fable 5.1 reaches only zero point five one recall at twenty, and that model may have seen the completed papers during its training.

Here the people who did the research mark the answer key, and the systems under test are asked to reproduce their hindsight from the question alone. Neither agentic search nor the strongest model named recovers much more than half of what the authors marked within the top twenty. The authors read this as a call for new training recipes that give models an expert's intuition for searching broad corpora, and they see the benchmark as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.