KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

KeyRec. Bounded Visual Memory for Streaming and Long-Video Understanding. By Zihan Chen and six others, from the University of Virginia and Google. Published on the twenty-sixth of September. No code repository is linked to it.

Vision-language models are increasingly asked to understand long videos and continuous streams. Dense visual tokens pile up as a video runs, which makes long-context inference prohibitively expensive. Training-free methods that select visual tokens cut that cost by keeping the informative ones. But they can lose coherent evidence about events, and they can fail to tell detailed recent observations apart from long-range history. Keeping every token costs too much, and keeping a flat selection risks losing the shape of what happened.

KeyRec is a training-free framework for building visual memory with a fixed bound. Its central move is to treat the recent past and the distant past differently. Fine-grained recent observations are kept in a visual cache. Historical evidence is organized into a structured event bank. Then, when a question arrives, a router decides how to split a fixed readout budget between the two. The bound is the point. However long the stream runs, the memory the model reads from stays within a fixed size.

Writing to memory happens without knowing the query. Candidate events are proposed according to how novel they are relative to events already stored, and the bank is maintained through an online update that adds, merges and evicts. Reading is where the question comes in. A text-only router adaptively allocates the fixed readout budget between recent memory and event memory, and does so without reprocessing historical frames. KeyRec operates on the visual embeddings the model actually receives, so it supports both modular encoder-projector VLMs and NEO-ov, an architecture with neither an encoder nor a projector.

The evaluation spans four streaming and long-video benchmarks and three VLM backbones. Using only ten percent of the dense decoder-facing visual-token budget, KeyRec achieves the best compressed performance in thirteen of fifteen settings.

On real-time questions it beats the strongest compressed baseline by between two point two one and eighteen point three seven points. It achieves the best compressed result in five of six long-video settings, and it performs best in every NEO-ov 2B setting.

Separation is the design choice that carries KeyRec. Recent detail and long-range history are different kinds of evidence, so they are stored differently, and the question decides how much of each to read. That is how memory stays bounded while a stream keeps running, without throwing away the structure of the events that came before.