VideoGen-Agent: Reinforcing Video Generation Agents

VideoGen-Agent. Reinforcing Video Generation Agents. By Binxu Li and eleven others, from Princeton University, Stanford University, and five other institutions. Published on the twenty-first of September. No code repository is linked to it.

Video generative models can now produce high-fidelity video that holds together over time. Some prompts still defeat them: those that need specialized knowledge, specific identities, physical consistency, or events that happen in a particular order. A single generation pass from the prompt alone often cannot satisfy that kind of request.

VideoGen-Agent treats video generation as a job for a multimodal agent with tools. Trained with multitask agentic reinforcement learning, it learns to use external tools of three kinds: augmentation, generation, and verification. It coordinates them over several turns, using the prompt and what it observes along the way to decide what to do next. Rather than asking one generator to get everything right in one attempt, the agent can prepare the request, generate, check the result, and act on what the check shows.

One shared policy is trained on a dataset balanced by category, covering six tasks. Training has two stages. Supervised fine-tuning on trajectories generated by a teacher establishes the basic tool-use behavior. Reinforcement learning then refines it. The reward is a category-aware hybrid, scoring three things: whether tool calls are valid, whether the tools used suit the task, and the quality of the video that comes out. The reward is not only about the final video, then. It also asks whether the agent went about the job in a sensible way for that category of task.

To test this, the authors introduce VABench, a held-out benchmark of six hundred prompts. It covers procedural knowledge, preserving the identity of one entity or several, physical consistency, scene composition, and temporal structure across multiple shots.

On VABench, VideoGen-Agent improves on its own base text-to-video generator by nineteen point one points, from fifty-six point five to seventy-five point six. Swapping in upgraded generation tools raises the score further, to eighty-six point one, with no additional training of the agent. And human raters prefer that upgraded configuration over the strongest standalone baseline in eighty-four point three percent of comparisons.

Consider the upgrade result. The agent learned how to use tools, not how to be a particular generator, so when a better generator was plugged in, the gains arrived without retraining. The authors read this as support for learning tool use across video-generation tasks, and as a sign that such an agent can benefit from later advances in the tools it calls on.