4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

4DCodeBench. Benchmarking Agents on Inverse Graphics of Dynamic Scenes. By Ruihong Shen and eight others, from Johns Hopkins University, Stanford University, and two other institutions. Published on the second of October. Its code repository is worked out: about one hundred source files, with a readme.

Inverse graphics means working backwards from what is seen to a program that would produce it. Doing that for a dynamic scene asks for more than geometry. An agent has to turn visual observations into a compact account of both the structure of a scene and how it changes, and that can mean implementing abstractions such as physical simulations to reproduce complex behaviour.

4DCodeBench measures exactly that capability. It is a benchmark for four dimensional inverse graphics through code generation: an agent watches a video and reconstructs the dynamic scene as an executable graphics program.

The material comes from two sources. The authors curate a set of real-world videos, and they construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow and fracture. The abstract does not give the number of scenes or the scoring details.

Frontier models were benchmarked extensively, and the headline result is a mismatch. Strong capability at static reconstruction does not yet translate into reliable reconstruction of complex dynamics. A model that can rebuild a still scene as code can still fail when that scene starts to move.

The benchmark is offered as a testbed for tracking progress toward agents that interpret the dynamics of the world through code, and it is publicly available. What it isolates is the step from describing how a scene looks to writing down how it behaves.