Memory as Weights: Internalizing Long-Term History for Streaming Videos

Memory as Weights. Internalizing Long-Term History for Streaming Videos. By Zhuoming Liu and four others, from the University of Wisconsin–Madison and Peking University. Published on the fifteenth of October. Its code repository is worked out: about ninety-five source files, with a readme and a licence.

Streaming vision-language models must keep integrating information from the past while staying responsive to current observations and to user queries. Existing memory mechanisms typically compress, retrieve or cache historical observations, and then reintroduce them into the input context. That forces historical memory and current observations to compete for the same limited context. The longer the stream runs, the more the past has to fight the present for room.

MaLoW moves history out of the context altogether. It is a streaming memory framework that internalizes historical context into evolving, lightweight parameter updates. History is represented as weights, while the current observation is kept as input. The context window is left to the present, and the past changes the model that reads it rather than taking up space in what the model reads.

A recurrent HyperNetwork incrementally consolidates incoming video observations into LoRA weights of fixed size. Historical information is carried forward in those weights, without the model having to revisit the full video stream. Each new stretch of video updates the weights, and the updated weights carry everything seen so far into the next stretch. The memory stays the same size however long the stream becomes. To support training, the authors construct MaLoW-Instruct, a large-scale streaming video dataset containing three hundred and eighty-one thousand question-answer pairs over fifty-six thousand videos.

Extensive experiments show strong performance on OVO-Bench, StreamingBench and StreamArena. On the backward split of OVO-Bench, MaLoW outperforms prior memory approaches by between three point nine percent and eleven point eight percent across different video backbones, while incurring only modest latency overhead. The framework also reaches beyond streaming video understanding. Applied to vision-language-action models, it improves the action success rate on Simpler WidowX by nine point three percent.

The division of labour carries the idea. The present arrives as input, as it always has. The past is written into a small set of weights that keep updating as the video runs, so the two no longer contend for one window. And the arrangement is not tied to answering questions about video: the same memory carried over to a model that acts, and helped it act.