CodeCap-Bench: Can Code Serve as a Good Video Caption?

CodeCap-Bench. Can Code Serve as a Good Video Caption?. By Jiafu Tang and thirteen others, from Nanjing University, Kling AI, and four other institutions. Published on the twenty-seventh of October.

A video caption is meant to keep the visual information of a clip in a compact description, and that description is almost always natural language. The trouble is precision. Saying exactly where things sit in space and exactly how they move over time tends to produce long captions, and sometimes ambiguous ones. The authors explore a different medium, which they call Code as Video Caption. Here an executable program describes the content, through graphical primitives, layout rules, and functions of time, as a complement to prose.

Before building a benchmark, they check that the idea is sound. Code captions and natural language captions are compared on two downstream tasks: answering questions about the video, and reconstructing the video. Code captions come out ahead on both, with better reasoning accuracy and better reconstruction fidelity. That moves the question elsewhere. The representation can work, but only if current multimodal large language models, or MLLMs, can actually write faithful code captions. CodeCap-Bench is built to measure that ability.

The benchmark contains three hundred and twenty-three videos, spread across five categories of structured two-dimensional content. Structured is the operative word: this is the kind of content that graphical primitives, layout rules and functions of time are suited to describe. Each video comes with fine-grained annotations, and models are scored through a multi-level evaluation framework rather than a single number, so a caption can be judged at more than one level of detail. The task for a model is to watch a video and write the program that captions it.

Extensive experiments give a split picture. The leading proprietary MLLMs show promising code captioning ability. Even so, they still struggle to capture complex temporal dynamics faithfully, which is exactly the part of a video that natural language was having trouble with in the first place. Open-source models lag substantially behind the proprietary ones.

The benchmark sits on a two-part argument. Code is shown to be a better container for what a structured video contains, and current models are shown to be only partway to filling that container well, with motion over time as the hardest piece to get right.