Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Code2Video-Compass. A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V. By Zhenghan Chen and four others, from Fudan University and Sun Yat-sen University. Published on the twentieth of October. Its code repository is public and real: seventeen source files.

There are two ways to get a video out of a model. One is to generate the pixels directly. The other is to have the model write a program, and let the program produce the video. The second route offers fine-grained control, targeted editing, and the reuse of components, but it asks a lot of the model. A multimodal large language model has to translate visual and temporal requirements into executable code that yields a complete video. The benchmarks that exist for this mostly target specialized tasks. Complete video creation, across diverse creative categories and across different kinds of input, has been left insufficiently evaluated.

Code2Video-Compass is built to fill that gap, and it covers three input settings. Text-to-video starts from a written instruction. Image-to-video starts from a picture. Video-to-video starts from an existing clip. What gets measured is whether a model, given any of these, can write the program that produces the video it was asked for.

The benchmark holds two thousand one hundred and four tasks. They span ten categories and twenty-four subtypes, and they are organised along three dimensions of complexity: construction, dynamic animation, and state dependency. Some tasks are curated from textual instructions and visual references. Others are derived from complete videos, both professional and user-created, turned into matching text, image and video tasks.

Scoring works at two levels. On the objective side there are twelve metrics, covering operational performance, video quality, text-video alignment, and fidelity to the reference. On the subjective side there are five more, where an MLLM judge assesses instruction following and the holistic quality of the video.

The evaluation covers fifteen recent models, proprietary and open-weight, drawn from nine families. Even the leading ones struggle to reliably generate high-quality complete videos that match both the textual and the visual requirements. The trend is consistent along the complexity axes: as construction, dynamic animation and state dependency grow harder, performance generally declines. The benchmark's own structure is what makes that visible, since each axis can be raised on its own.

The authors hope the benchmark will push code-based video generation toward being reliable, coherent and controllable, and encourage wider progress in programmatic visual creation. Its contribution is a common yardstick for an approach whose appeal is control. Writing a program for a video promises precise editing and reuse, and Code2Video-Compass measures how far today's models are from delivering a whole video that way, and where along the three axes of difficulty they begin to fall short.