rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

rTaste. Do Research Agents Have Taste? Evaluating Early Stage Research Judgment. By Art Kanke, from the University of Minnesota. Published on the twenty-ninth of October.

Research agents can propose experiments. The harder question is whether they have taste, meaning the judgment of which experiments deserve to be run before anyone knows how they will come out. That judgment is exercised early, when outcomes are still unknown, and it is the part of research that is hardest to check after the fact, because the experiments nobody chose are never run.

rTaste measures three things an agent does at that early stage. It proposes minimum viable experiments, or MVEs. It forecasts whether each one will pass a threshold that the agent sets for itself. And it ranks them. Each of those can then be held against what actually happens: whether the proposals are good, whether the forecasts are right, and whether the ranking puts the best experiment first.

The setup gives six frontier models seven recent NLP papers. Each model works as a research agent, either alone or as part of a team, and produces its proposals, forecasts and ranking. The design choice that makes the evaluation possible comes next. A separate executor attempts every MVE, including the ones the agent ranked last, so that a low ranking does not hide the result. Three model auditors then judge the outcomes.

Of four hundred and fifteen MVEs, one hundred and ninety-four are judged sound, valuable and new. Fewer than half of those, seventy-three, are also judged to go beyond the seed paper and to be non-obvious. On ranking, the agents' first choices match the auditors' most valued experiment more often than a random pick would, but they are not shown to improve on simply choosing the experiment that was written first.

Forecasting looks similar. The agents' forecasts beat a base rate, yet they are not measurably more accurate than those of a fresh instance of the same model given only the paper and the registrations. Teams cost more, without a clear overall improvement in proposal quality. A dataset of all four hundred and fifteen MVEs, with their forecasts and audited outcomes, is released with the evaluation harness.

Back to the question in the title, then. The answer the evidence supports is split. These agents generate plenty of experiments the auditors value. Where the evidence stays limited is the step that taste is really about: using their rankings to decide which of those experiments to run.