VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents
Jinfa Huang, Jianming Xu, Jingyang Lin, Zhengyuan Yang, Jiebo Luo
University of Rochester · Microsoft
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 11 · arXiv 2609.38119
The abstract
Long-form video understanding requires multimodal agents to iteratively gather evidence over many reasoning steps. However, most existing agentic methods suffer from semantic thrashing: as append-only working memory grows, attention to key evidence collapses, and the agent loses access to what it has already found. First, we provide a structural argument showing that append-only memory can incorporate newly observed target evidence, but cannot remove accumulated noise or prevent ordered context growth without a rewrite operator. Second, motivated by this analysis, we propose VideoLoop, a multimodal agent with two coupled loops. The outer loop reasons over the video and the inner loop, after each step, retrieves artifacts from an unbounded filesystem of past observations and intermediate analysis, and rewrites a bounded working memory. Extensive experiments demonstrate the effectiveness of VideoLoop, which improves four popular LVLM backbones in a plug-and-play manner, with an average gain of 4.2% points over baseline on VideoMME (long). Further analysis of working memory suggests that VideoLoop mitigates semantic thrashing: on the hardest quarter of VideoMME (long) questions, a blind judge that reads only the agent's context answers 81.1% correctly, versus 60.9% for the append-only agent. With Gemini 3.1 Pro, VideoLoop reaches 88.3% on VideoMME (long), 88.8% on VideoMMMU, and 80.9% on LongVideoBench (long).
The Brief
2:18A short podcast summary of the paper. Written and voiced by AI.
Transcript
VideoLoop. Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents. By Jinfa Huang and four others, from the University of Rochester and Microsoft. Published on the twenty-ninth of September. Its code repository is public and real: about thirty source files.
Understanding a long video means gathering evidence over many reasoning steps, and multimodal agents do it iteratively. Most existing agentic methods keep a working memory that only ever grows: each step appends to it. The authors name the resulting failure semantic thrashing. As the append-only memory grows, attention to the key evidence collapses, and the agent loses access to what it has already found. The clue was gathered; it is simply buried.
The paper first makes a structural argument. Append-only memory can take in new target evidence when it is observed. What it cannot do, without an operation that rewrites it, is remove the noise that has piled up, or stop the context from growing in order. Some way of rewriting memory is therefore needed, and VideoLoop is built around one.
VideoLoop is a multimodal agent with two coupled loops. The outer loop reasons over the video. The inner loop runs after each step of the outer one. It retrieves artifacts from an unbounded filesystem that holds past observations and intermediate analysis, and it rewrites a working memory that is kept bounded. Nothing is thrown away, since the filesystem keeps it all, but what the agent actively carries stays small.
Plugged into four popular vision-language backbones, VideoLoop improves each of them, by an average of four point two points over baseline on the long split of VideoMME. A closer look at the working memory supports the thrashing diagnosis. On the hardest quarter of those long questions, a blind judge that reads only the agent's context answers eighty-one point one percent correctly, against sixty point nine percent for the append-only agent.
Paired with Gemini 3.1 Pro, VideoLoop reaches eighty-eight point three percent on long VideoMME, eighty-eight point eight percent on VideoMMMU, and eighty point nine percent on the long split of LongVideoBench.
The blind judge connects the numbers back to the argument. Reading only what the agent is holding, it is asking a narrow question: is what is needed still there to be found? For a memory that gets rewritten, the answer is yes far more often than for one that only grows, which is the gap the structural argument predicted.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The agent found the clue. Then it buried it under everything else it saw.
You are building an agent that answers questions about hour-long videos. It looks at one stretch at a time, adds what it saw to its notes, and decides where to look next.
The road ahead
- 1 · The problem ~6 min An agent watching a long video, whose notes only ever grow ends up burying the clue it already found under piles of noise.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Keep everything in an unbounded store, and after each step rewrite a small working memory from what the question needs.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each step: observe a video region, save findings to the filesystem, retrieve what the question needs, rewrite the bounded memory.
- 6 · The experiment ~6 min You’ll guess: Plugged into four vision-language backbones, how much did VideoLoop improve accuracy over baseline on the long split of VideoMME, on average?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.