UNREAL: Unifying Retrieval and Long-Context with a Single Model

UNREAL. Unifying Retrieval and Long-Context with a Single Model. By Edan Kinderman and six others, from NVIDIA and the Technion. Published on the sixth of October. No code repository is linked to it.

Long-context inference and Retrieval-Augmented Generation both have to pick out the evidence that matters, but they do it at very different scales. One works inside a single long prompt. The other searches an entire corpus. They are usually treated as separate problems with separate machinery. The question here is whether one mechanism, living inside the model itself, could select evidence across that whole range.

UNREAL, which stands for UNifying REtrieval And Long-Context with a Single Model, is a model-native evidence selection framework meant to span both corpus retrieval and long-context inference. Instead of bringing in a separate retriever, it relies on the frozen LLM's own internal representations.

Chunks are encoded, and retrieval queries are derived, directly from those internal representations. The addition is small: fewer than five hundred thousand trainable parameters, with the backbone left unchanged. The same selection mechanism then serves both jobs. For retrieval it searches an index. For long-context tasks it removes distractors before generation begins.

Retrieval was tested on a Wikipedia index of three billion tokens and twenty-one million chunks. All four UNREAL backbones, dense and hybrid, outperform state-of-the-art retriever and reranker systems. The best model raises recall on HotpotQA from forty-nine point one percent to seventy-three point two percent. On 2WikiMultiHopQA it goes from thirty-one point seven percent to sixty point one percent. And on MuSiQue, from eight point eight percent to fourteen point four percent.

On long-context tasks, removing distractors raises NoLiMa accuracy from one percent to twenty-four point eight three percent, at that benchmark's maximum context length of a hundred and twenty-eight thousand tokens. On LV-Eval, at two hundred and fifty-six thousand tokens, the F1 score moves from forty-nine point nine seven to fifty-four point six six. There is a cost benefit too. Relative to full-context inference, UNREAL reduces FLOPs and time to first token from roughly thirty-two thousand tokens onward, and the gains grow as the context grows.

The authors read these results as establishing model-internal evidence selection as a common foundation for two tasks usually built apart: corpus retrieval, and long-context inference where the evidence is sparse. A frozen model, a very light addition, and one mechanism doing the choosing at both scales.