Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Rare, Relevant, and Still Wrong. How Memory Fails LLM Group Decisions. By Anouar Imel and three others, from the University of Electronic Science and Technology of China, Xidian University, and two other institutions. Published on the twenty-seventh of October.

Picture a group of LLM agents discussing a problem, with their conversation compressed into a short memory that a final decision maker reads. What should that memory keep? Two heuristics come naturally. Keep the rarest claims, or keep the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is tangled up with whether it helps. The claim of this work is that both heuristics fail, and in predictable places. Accuracy tracks the balance of evidence the memory ends up keeping, not the policy that chose it.

To untangle the confound, the authors build CrossProfile, a diagnostic benchmark of sixty templates with answer keys that are verified automatically. In it, who owns the supporting facts, the decoys and the irrelevant facts is crossed, while the wording of those facts stays fixed. That yields four hundred and eighty tasks. The models tested are Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models.

Selecting by relevance lets decoys back in, because decoys are on topic. Even a perfect relevance filter loses between twenty-three and thirty-one points to rarest-first selection in the classic hidden-profile configuration. Rarest-first, in turn, is no better than random selection on average. It wins only where the rare facts happen to be the useful ones.

The decision model adds a third failure of its own, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random while keeping line order fixed, raises accuracy in the classic configuration by between thirteen and twenty-four points. And in natural discussions the evidence is gone before any memory gets to act. The complete transcript produces lower accuracy than having no notes at all.

These results come from a controlled diagnostic. The facts are templated, their text is held fixed, and ownership is varied by design, which is what lets rarity and usefulness be separated. The point gains are reported for the classic hidden-profile configuration. The model set is four models, two of them from DeepSeek, and the natural-discussion result is a separate observation about transcripts rather than about any memory policy.

There are three ways the pipeline breaks, then. A relevance filter cannot tell a decoy from support. A rarity filter is only as good as the coincidence that rare facts are useful. And the reader of the memory counts heads. Each failure comes down to which evidence survives and how the decision maker weighs it.