Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

Length Is Cheap, Competition Is Not. Attention Predicts Context Rot in Retrieval. By Yinuo Liu, from the University of California San Diego. Published on the third of November.

Why do language models fail on long inputs? The usual answer is length. This work argues that, for retrieval at least, length is cheap and competition is not. What hurts a model is not how much text surrounds the evidence, but how much of that text competes with it for attention.

The tests run on five open models from four families, ranging from zero point five billion to three billion parameters. Neutral filler costs nothing: accuracy stays between zero point nine seven and one, up to thirty-two thousand tokens on one of the models. Swap that filler for statements written in the format of the evidence, placed at the same positions, and accuracy drops to between zero point one six and zero point three three. A single statement that gives the queried key a rival value multiplies the loss by anywhere from nine to one hundred and ten, and each additional rival adds to it.

Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across one hundred twenty-six thousand four hundred seventy-three trials, it explains up to fifty-eight percent of the variance in loss. Log length explains at most eight percent. The author also revisits the masked-filler protocol behind the claim that length alone hurts. On MMLU it reproduces, with eleven to eighteen points lost at thirty thousand tokens. In retrieval, though, it fails a visible control. Masking costs forty-five accuracy points at thirty-two thousand tokens, while the same filler left visible costs none.

The claim is stated for retrieval, and the models are small: five open models of up to three billion parameters. The masked-filler result on MMLU is reported as reproducing, so the argument is not that length never matters anywhere, only that in retrieval the loss follows competition for attention rather than length itself. The probe and all one hundred twenty-six thousand trial-level measurements are released.

In retrieval, then, context rot is a competition for attention. Filler that looks nothing like the answer is harmless. A distractor that looks like the answer, and especially one that offers a rival value for the same key, is what pulls attention away, and the attention left on the evidence is what predicts whether the model gets it right.