BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

BVB. Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender. By Yolo Y. Tang and sixteen others, from the University of Rochester, Sony Group Corporation, and two other institutions. Published on the fourteenth of September. Its code repository is public and real: about thirty source files.

Multimodal agents can create complex videos in software such as Blender by writing code, without relying on diffusion models. Video understanding benchmarks, though, still evaluate models mainly through question answering. The premise here is different: if an agent truly understands a video, it can reconstruct that video programmatically. Understanding is tested by rebuilding rather than by answering. Question answering checks whether a model can report facts about a video. Reconstruction asks for more: the agent has to produce the video's content itself, as code, and the result can be compared with the original.

BVB, short for Blender-VideoBench, tests that ability by asking agents to reconstruct real-world videos as animated Blender scenes. The benchmark renders each reconstruction from its animated camera and evaluates it on two axes. Dual VQA measures how many spatiotemporal facts the reconstruction preserves. Latent Similarity measures how closely the reconstruction matches the source video perceptually. The overall score is a square-root mean, which favours balanced performance, so a reconstruction cannot score well by excelling on one axis and neglecting the other. The two axes separate a video's meaning from its look. A reconstruction could match the look while getting the events wrong, or preserve the events in a scene that looks nothing like the source.

To ensure a fair comparison, every agent programs its reconstruction through the same lightweight harness, Mini-BVB, in an identical sandbox, under a shared cost limit. The evaluation covers fifty-one configurations from ten model families, and analyses semantic retention, perceptual similarity, reasoning effort, and cost. With the cost limit shared, no configuration can pull ahead simply by spending more.

The best model reaches eighty-eight point six on Latent Similarity, yet retains only fifty-three point seven percent of the source-correct spatiotemporal answers. Additional reasoning improves visual similarity, but it does not close this gap in factual accuracy. In a blind study with fifteen raters and five configurations, Latent Similarity correlates strongly with human preference, which supports using it as the perceptual axis. More reasoning effort buys better-looking reconstructions rather than more accurate ones.

A reconstruction can look right and still get the facts wrong, and on this benchmark that is the pattern at the top: the scene resembles the video while losing close to half of the spatiotemporal facts. The authors conclude that programmatic reconstruction is a viable test of agentic video understanding, and that semantic retention remains the main challenge. The benchmark separates two things a question-answering test would blend together.