IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

IdeaTaste. Do LLMs Share the Research Taste of Human Researchers?. By Shanshan Zhong and five others, from Carnegie Mellon University and the Mohamed bin Zayed University of Artificial Intelligence. Published on the second of November.

Research taste is the judgment behind research: what to read, which problems to pursue, which methods to try. As AI scientists take on more of the work, that judgment shapes more of what they do. Yet it is far less understood than their ability to carry research out. Carrying out a plan well and choosing the right plan are different skills, and the second has had much less attention.

IdeaTaste measures taste against what actually happened next. An LLM is given a snapshot of the literature, frozen in one year, and writes research proposals from it. Those proposals are then scored by how closely they match the work the same research communities went on to publish the following year. The model cannot see that later work. It has to anticipate it, the way a researcher reading the field today has to guess where it is going. Matching is checked in two ways: on the literature a proposal cites, and on the ideas it states. Each score is read against the human level, the community's own published work, so the question is always how far a model's judgment sits from the people actually doing the research.

The experiments cover six contemporary language models and four research areas, ranging from established fields to emerging ones. Altogether the proposals number in the tens of thousands. Two further ingredients sit alongside the automatic scoring. The authors vary how much field context the models get, and how large the populations of agents are. And they run a blinded expert review that sets AI proposals side by side with the papers the communities actually published, without telling the reviewers which is which.

LLM agents fall well short of human communities on both counts. On citation matching they reach about twenty-six percent of the human level. On idea matching they reach only eighteen percent. The shape of the shortfall is specific. Populations of agents can cite as diversely as a community does, and they take up its tasks and methods. What they rarely reach are the concrete approaches and research plans the community actually pursues. Neither richer field context nor larger populations substantially closes the gap.

The expert review points the same way. AI proposals trail the real papers least on presentation and most on expected success, which mirrors the gap in concrete approaches. And the proposals whose ideas no human paper shares do worst of all. That suggests they are weaker directions, not opportunities the humans overlooked.

The authors' reading is that the main bottleneck in AI research taste is not finding the directions a community is working on. It is turning those directions into the concrete approaches the community goes on to pursue. The models seem to know where the field is heading, roughly, but not what it will actually do when it gets there. IdeaTaste is released as an open leaderboard that is continuously updated, so contemporary AI proposals can keep being checked against publication cycles that have not happened yet.