Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models

Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models. By Taewhoo Lee and eight others, from Korea University, Amazon, and three other institutions. Published on the twenty-first of October.

Search agents let large language models take on complex tasks by retrieving external information and synthesizing it. Many of them, though, lack what the authors call search-state awareness. That is the ability to recognize what information has already been obtained and what is still missing. Without it, two failures follow: reasoning that hallucinates the missing pieces, and searches that stop before they should.

Self-Assessment Reinforcement Learning, or SARL, teaches that awareness during training. Models learn to recognize their own search state as they train, and to improve their search behaviour accordingly. The important constraint is that none of this needs an extra assessment step at test time. The awareness is meant to be learned into the model rather than bolted on around it.

The mechanism is a dual reward. One part captures the corrective effect of each self-assessment: the difference an assessment makes in setting the search back on course. The other penalizes excessive reliance on assessment. The two pull in different directions on purpose. The first rewards checking where checking helps; the second keeps the model from checking all the time. Between them, the design jointly encourages search trajectories that are both effective and efficient.

In this setup SARL achieves a twenty-four point three percent relative improvement over the strongest baseline. Against the strongest baseline for each individual benchmark, its average gains are thirty-five point nine percent on multi-hop reasoning and twenty-seven point nine percent on deep research benchmarks. It also surpasses methods that rely on auxiliary supervision or on coordinating several agents, even though SARL is a standalone reinforcement learning framework with no scaffolding at inference time.

There is a result beyond final performance as well. SARL keeps training stable under extended search horizons, meaning longer searches with more steps, a regime where existing methods consistently degrade. That stability is what lets its gains be sustained as task complexity grows, rather than falling away on the hardest problems.

Knowing what you still do not know is the skill at stake here. SARL's bet is that an agent can learn it during training, through a reward that values useful self-checks and discourages leaning on them, and then search well on its own when it is deployed.