VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation

VibeVideoBench. Evaluating Multimodal Agents on Code-Driven Video Generation. By Muxi Diao and twenty-three others, from the Beijing University of Posts and Telecommunications, the Zhongguancun Academy, and three other institutions. Published on the twenty-seventh of October.

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a game that stays consistent with its rules, or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. Each of those failures is a matter of exactness. A formula has to be right to the symbol, a chart has to be correct, a game has to obey its rules, and narration has to land on the event it describes. Writing the video as a program is the route offered for that kind of precision, and a route like that needs its own yardstick.

VibeVideoBench evaluates agents on code-driven video generation. Each task asks an agent to deliver a finished MP4, with sound, within one hour. The agent runs in its native harness inside a sandbox, with no image or video generation model available, so everything on screen and in the soundtrack has to come from what the agent builds itself. The hour is part of the test, not a convenience. An agent is measured on delivering a finished video, so running out of time counts.

There are two hundred and thirty-nine self-contained tasks, in English and Chinese, spanning twelve categories, from data stories and music-driven pieces to classroom lessons and paper explainers. Evaluation is hierarchical and evidence-based. Program checks verify the delivery spec. About thirteen thousand task-specific rubric checks are each traced to the requirement they enforce. Five quality dimensions, anchored per task, are scored by three panels of VLM judges. The layers run from the strict to the judged. A program check either passes or fails, a rubric check points back to a specific requirement, and only the quality dimensions call for judgement, which is why they are anchored to each task.

Nine frontier agent systems were evaluated, across two thousand one hundred and fifty-one scored rollouts. GPT-6 Astra and Claude Opus five point five rank first and second, with the judges and with human annotators alike. The strongest system satisfies every core requirement on seventy-six point three percent of tasks. More broadly, the judges' ranking of the systems closely matches blind human preferences, which is what lets the automatic scores stand in for people across thousands of rollouts.

Delivery is a second axis. Counting missing videos as zero changes the ranking. Three quarters of the lost videos are final renders that started too late, ran too slowly or never started at all, which the authors describe as a failure of how agents plan their hour. And although agents build their own rendering pipelines and inspect their own frames, how much they look does not predict quality.

Two separate questions sit inside every score here. One is whether the video is exact and complete. The other is whether a finished file exists at all when the hour runs out, and on that second count the abstract places most failures in planning rather than in rendering.