Video-Coder: Reinforcing Agentic Coding for Video Generation

Video-Coder. Reinforcing Agentic Coding for Video Generation. By Chaoyang Wang and nine others, from the University of Central Florida, the University of Würzburg, and four other institutions. Published on the sixth of November.

Video generation has improved remarkably in visual quality and in temporal consistency. Three things remain hard: getting the spatial structure precise, keeping the motion dynamics coherent, and controlling how a scene is composed. Those are properties of arrangement and movement, more than of how each frame looks. Progress on the look of a video, in other words, has not by itself delivered precise control over where things are, how they move, and what the scene is made of.

Video-Coder approaches them through code. The authors describe it as the first attempt to train a code-driven video generation agent. The agent writes code and executes it, autonomously, to build a dynamic 3D scene. That scene, rendered out, is called the Draft Video, and it hands structure and motion signals to a downstream video generator, which does the rest. The agent's output, then, is not the finished video. It is an intermediate: a constructed scene whose job is to carry structure and motion, handed to a separate model that produces the video people actually watch.

Training an agent like this needed its own supporting pieces. A dedicated data pipeline produced two curated datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, which hold diverse prompts together with ground-truth reference videos. Alongside them sits the Video-Coder Harness, a single interactive environment that combines code execution, 3D scene rendering and execution feedback, and is used for both training and inference.

Training itself has two stages. Supervised fine-tuning comes first. Then comes agentic reinforcement learning that runs natively inside the harness, with a dual reward: one part based on execution, one part based on the video. An annealing strategy gradually reduces the execution reward over time, so that in the later stages of training the model relies more on the video reward.

The experiments report significant gains for Minimax-H3 when it is paired with Video-Coder. Its score improves by approximately seven points on PhyGenBench, and by approximately twenty-four points on VBench-2.0. Note which model those gains belong to: Minimax-H3 is the downstream generator, and what improves is its output once Video-Coder supplies the draft. The authors say they fully open-source the data and the models, and offer the work as an open foundation for coding agents in video generation.

What stands out is a division of labour. Visual quality is where video generation has already come far; structure, motion and composition are where it still struggles. Video-Coder hands that second set of jobs to an agent writing code for a 3D scene, and leaves the generator to work from the draft that scene provides. The reward schedule mirrors that order: first learn to make code that runs, then learn to make code whose videos are good.