When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model

When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model. By Rishabh Sharma and one other, working independently. Published on the twenty-eighth of September. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

Does conversational memory need facts extracted by a language model, or is it enough to select the right raw turns? Published results disagree. Extraction-based systems report gains from distilled facts, while recent studies find that raw history with good ranking does as well, and then disagree about whether ranking matters. This pre-registered study finds that at a tight budget, selection can stand in for extraction. It also offers a reason for the disagreement: what reranking is worth depends on the budget.

The study uses held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns chosen by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory. The one-sided ninety-five percent bound is minus three points, inside a margin of minus five points. Blind human grading narrows the margin but does not change the result. Raw turns are three thousand and sixty-one times cheaper to write, and the result holds with a second answer model.

Reranking carries the budget story. When three of thirty candidates are kept, reranking adds seventeen point four points on LoCoMo, and nine point one on LongMemEval. At generous budgets it adds only one point five and one point one, and there the extraction systems are more accurate. A gain that shrinks as the budget grows is one way studies run at different budgets could reach different conclusions.

At matched context, Jev selects as accurately as an LLM reranker, with a non-inferiority bound of minus two points, at a third of the latency. It is also more accurate than a graph traversal that makes multiple calls. One cost of reranking is noted: it lowers correct abstention.

The conclusions hold within this study. Non-inferiority was shown at a tight budget on LoCoMo. At generous budgets, extraction systems came out ahead, so the claim is not that extraction is never needed. The shrinking value of reranking is observed across the budgets tested here, on these two datasets, and the authors put it as suggesting why published results disagree rather than settling the matter. Plans, code and graded answers are released, which is what lets a pre-registered result be checked against what was planned.

The useful frame is a budget, not a verdict. When context is tight, picking the right raw turns with one cheap call matches extracting facts. When context is generous, ranking adds less and extraction pulls ahead. Where a published result sits on that axis may explain which side of the debate it lands on.