Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

Video-RSI. Recursive Self-Improvement of Video Understanding Agents via Harness Evolution. By Bingjun Luo and two others, from Tsinghua University and Harbin Engineering University. Published on the twenty-ninth of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.

A video understanding agent gathers its evidence through an executable harness. The harness controls what the agent observes and how it uses those observations. When the agent fails, the natural place to look for a fix is the execution trace. But a trace only contains the evidence that the current harness happened to acquire. Several explanations for the failure may fit that trace equally well, and nothing in it can tell them apart. That leaves a thin basis for any attempt at self-improvement.

Video-RSI is a framework for recursive self-improvement in which the agent uses its own language model to revise its own harness. The step that matters is going back for more evidence. Rather than reasoning only over the trace it already has, the model returns to the video to test which explanation of the failure is actually right.

That return trip is called active video investigation. The model revisits the original training videos and makes additional observations aimed at the competing failure explanations, so any change it proposes is grounded in evidence beyond the existing trace. A second component, cost-aware harness evolution, turns those diagnoses into reusable revisions of the harness. It also decides which revisions to keep, weighing two things together: answer accuracy, and visual cost, meaning how much the agent has to look at.

Across the authors' evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames. Against existing video understanding agents, it reaches competitive trade-offs between accuracy and efficiency. The authors read this as a demonstration of the potential for such agents to improve their own evidence acquisition and use. The code is public.

What sets this apart from ordinary self-reflection is where the agent looks when it reflects. A trace records only what was seen. By going back to the video itself, the agent can settle questions its own record could not, and by counting frames as a cost, it is pushed to improve by looking better rather than simply looking more.