Video Generation Meets Agent Harness: Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement

Video Generation Meets Agent Harness. Agentic Video Generation, Agentic World Modeling and Recursive Self-Improvement. By Liaoyuan Fan and twenty others, from Video Rebirth, the National University of Singapore, and five other institutions. Published on the twenty-ninth of October.

Two shifts are happening around video models at the same time. Video generation is moving away from end-to-end synthesis and toward agentic generation, where planning, tool use and feedback produce multi-shot videos that are coherent and controllable. Meanwhile, video world models supply action-conditioned observations and predicted futures, which makes them interactive environments that agents can act in. This report studies where those two meet, and it names the meeting point the agent harness.

A harness, as defined here, is an executable runtime wrapped around video models. It constructs context, exposes tools, maintains state, routes feedback and manages budgets. The video model generates. The harness decides what the model sees, what it can call on, and what happens with its output.

The report is organised along three directions. The first is agentic video generation. Here harnesses coordinate scripting, shot planning, asset management, production and editing, and the report also takes up how such systems should be evaluated, which is itself a challenge.

The second direction is agentic world modeling, where the harness mediates observation, control, execution and state. The authors discuss hybrid designs in which a game engine or an agent-written program keeps the world state while a video model renders the observations. They also discuss how agents and world models can evaluate and improve each other.

The third direction is recursive self-improvement in how video models are developed. The report separates the levels at which improvement can happen: revising outputs, updating harness state, updating generator parameters, updating evaluators, and changing the improvement procedure itself. It then traces how those improvements feed back into later development.

Cutting across all three directions is a set of agent capabilities the report treats as essential. Audiovisual perception. Long-horizon planning. Memory. Code-based interaction. Feedback-guided control. Skill reuse. And coordination among multiple agents. The same harness ideas recur whether the goal is a finished multi-shot video, a simulated world, or a better model.

The open challenges are gathered at the end: autonomous video production, lifelong memory, programmable hybrid control, autonomous model development, provenance, and cost accounting. The last one ties the rest together. It is evaluation that links video quality and task performance to sustained improvements in both the models and the harnesses around them.

The report's central move is a change of unit. It looks at the runtime that surrounds the video model, the thing that plans, remembers, calls tools and judges results, rather than only at the model itself. Seen that way, video production agents, world models and self-improving pipelines become three uses of one structure, and the questions about how to evaluate them become shared questions too.