Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads

Agentic RAG Evaluation. Budget Allocation Across Questions, Trajectories, and Reads. By Jingjie Ning and two others, from Carnegie Mellon University. Published on the fourth of October. No code repository is linked to it.

Evaluating agentic retrieval-augmented generation costs a budget, and that budget can be spent three ways: on more questions, on more search trajectories per question, or on more repeated answers, called reads. The central finding is that breadth wins. At a fixed spend of about thirty-four point four million model tokens, covering more questions lowers the standard error by thirty-three percent compared with five reads, and by twelve point six percent compared with three trajectories.

The measurements come from a retrieval-feedback comparison on HotpotQA and MuSiQue, tracking allocation precision, reading efficiency, and cost boundaries. Archived forecasts predict these allocations closely: the nested forecast within four percent, and the Q-only forecast within three point five percent. Depth subsets show no clear forecasting advantage beyond the two-trajectory audit.

Settling for a single read costs between zero and nine point nine percent in variance, relative to the fitted optimum at the same token budget, with substantial uncertainty for Pro. Setting temperature to zero cuts answer disagreement from fourteen point three percent to three point four percent, while comparison precision stays about the same.

The money picture is only partly settled. Under the recorded model fees, more questions beat more trajectories when search costs between nothing and one dollar per one thousand requests. Whether more questions beat more reads on fees remains unresolved.

For anyone sizing an agentic RAG evaluation, the advice is about where the marginal token goes: toward a wider set of questions before repeated passes over the same ones.