Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

Beyond Retrieval. Progressive Latent Memory Evolution for Streaming Video Understanding. By Hongyu Qu and nine others, from Nanjing University of Science and Technology, Ant Group, and two other institutions. Published on the third of September. No code repository is linked to it.

Streaming video understanding asks a multimodal model to process continuous visual input and answer questions about it under two hard constraints: strict causality, so it cannot look ahead, and bounded memory, so it cannot keep everything. The standard answer is to compress what has been seen into an external memory bank, then retrieve whatever looks relevant to the query and hand that back as extra visual context. It works. But the historical evidence stays external the whole time, sitting outside the model as context to be looked up rather than becoming part of how the model is reasoning.

LatentStream shifts streaming memory from store-and-retrieve to retrieve-and-internalize. The distinction is where the history ends up. Instead of remaining a bank that gets queried, retrieved evidence is folded into a compact, evolving latent memory of fixed length, which then continuously guides the reasoning as the stream continues.

Three coordinated components do this. Query-agnostic Hierarchical Streaming Memory organizes visual history into short-term, mid-term and long-term levels under a fixed memory budget, using Jenks-guided adaptive consolidation, and it does this before any query arrives. Once a query does arrive, Hierarchical Latent Memory Evolution takes over: groups of latent memory tokens are given progressively expanding memory receptive fields, so each group retrieves historical evidence from its own scope and internalizes it, iteratively, into that fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization builds a hierarchical progression reward from group-wise predictive entropy, and refines the latent memory tokens and the retrieved evidence together, pushing toward increasingly confident reasoning as the stream runs on.

The reported result is new state-of-the-art performance across existing online and offline video benchmarks. Both halves of that are worth separating. Online is the setting the method was designed for, where causality and a memory budget bind. Offline is the setting where a model may see the whole video and the constraints that motivated the design do not apply, so a method built for streaming has no structural advantage there and could reasonably have lost ground. Holding both is the stronger version of the claim.

The architectural claim underneath is that retrieval was being asked to do a job it is not shaped for. Looking something up and having absorbed it are different states, and only the second one compounds as a stream gets longer.

Beyond Retrieval. Progressive Latent Memory Evolution for Streaming Video Understanding. By Hongyu Qu and nine others, from Nanjing University of Science and Technology, Ant Group, and two other institutions. Published on the third of September. This is the long version: the whole paper rather than the abstract.

Streaming video understanding is the setting where a model watches continuously and you can ask it something at any moment. Live monitoring, driving, smart glasses, robots. Two constraints define it and neither is negotiable: the model cannot look ahead, and it cannot keep everything it has seen. Something has to be thrown away, and the question is what the survivors are for.

The dominant approach compresses history into an external memory bank, then retrieves query-relevant evidence when a question arrives and hands it back as extra visual context. The most appealing versions are training-free, adapting off-the-shelf video models to streaming without touching their weights, using frame selection, token pruning, cache management, and hierarchical memory. It works. The paper's objection is about where the evidence ends up. Retrieved evidence stays external the whole time, presented to the model as variable-length visual context to be read. So the paradigm answers which history should be accessed, and leaves untouched how the accessed history could become a compact state that participates in the reasoning directly. Query-agnostic memory and query-conditioned reasoning stay loosely coupled, with nothing in between that accumulates and evolves.

LatentStream turns store-and-retrieve into retrieve-and-internalize, through three components that run in order. The first is query-agnostic: hierarchical streaming memory organises arriving observations into short, mid and long-term levels under a fixed budget, using an adaptive consolidation strategy to shed temporal and spatial redundancy while keeping informative evidence. This happens before any question is asked, which is what makes it usable in a live stream. The second starts when a query arrives. Latent memory tokens are split into three groups with progressively expanding memory receptive fields, so each group can see further back than the last. At each iteration the tokens retrieve evidence from their own scope and evolve together with what they retrieved, and that evolved state then guides the next round of access. Retrieval and internalization take turns rather than happening once. The third component regulates all this. It builds a progression reward from the predictive entropy of each token group, on the reasoning that as the accessible historical scope expands, the model's confidence ought to increase. The latent tokens are then optimized against that objective at test time.

Five benchmarks, spanning both online and offline settings. On OVO-Bench it reaches sixty-four point two, on StreamingBench seventy-six point nine, and on the offline long-video side sixty-six point six on VideoMME, seventy-four on MLVU and sixty-two point one on LongVideoBench. The comparison that carries the argument is the same backbone with and without it: on OVO-Bench, adding LatentStream to a seven billion parameter model moves it from fifty-four to sixty-four point two, a gain of ten point two points, and the smaller three billion version gains six point eight. Underneath that average sits one result that deserves isolating. On the backward-tracing highlight detection task, the bare backbone scores twenty-three point seven and LatentStream scores sixty-eight point three. That is the task most dependent on having genuinely retained something about earlier video rather than about the recent window, which is precisely what the method claims to do.

There is no limitations section here at all, which is a gap to name rather than quietly work around. What can be read off the evaluation is that everything runs at one frame per second, on a three billion and a seven billion parameter backbone, and that the offline gains are more modest than the streaming ones, which is what you would expect from a method designed for a constraint that offline evaluation does not impose. The test-time optimization is an inference-time cost that the accuracy tables do not price, and no ablation of that cost is reported alongside them.

The architectural claim underneath is that retrieval was being asked to do a job it is not shaped for. Looking something up and having absorbed it are different states, and only the second one compounds as a stream gets longer. Whether the latent tokens are really internalizing anything, as opposed to acting as a very good learned summary, is not something the benchmarks can settle. But the highlight-detection jump is the kind of evidence that would be hard to produce by accident.