LEGO-Anything: Coding Agents for 3D Scene Reconstruction

LEGO-Anything. Coding Agents for 3D Scene Reconstruction. By Xirui Li and nine others, from the University of Maryland, College Park and AWS. Published on the twenty-eighth of September. No code repository is linked to it.

A 3D scene reconstructed from a single image is most useful, the authors argue, not as a rendering or a fixed 3D output, but as an explicit scene program. Run the program and you get a scene that can be inspected, edited and queried. LEGO-Anything is their Image-to-Code framework for producing such programs. A coding agent iteratively writes and executes Blender code, inspects the scenes and renderings it gets back, and revises the program. What that approach needs, and what the paper supplies, is a way to evaluate scene recovery end to end, from the photograph to the rebuilt scene.

LEGO-Bench is that evaluation. It scores three things separately. First, whether the artifact the agent delivers is valid at all. Second, the geometry of the visible surfaces. Third, the rendered appearance. Keeping those apart is what lets the benchmark say where an agent falls short, not only that it does. A program that runs cleanly but builds the wrong room should not score like a program that builds the right room, and a room with the right shape but the wrong look should not score like either. Three separate scores make those cases distinguishable.

The benchmark is simulator-grounded, with two hundred and eight images drawn from one hundred and four diverse scenes, indoor and outdoor. Grounding it in a simulator makes it extensible and allows precise automatic evaluation. A second test, called LEGO-World, asks whether reconstructed scenes can stand in for natural images. Object detections, instance masks and relative depth are derived as deterministic queries on scenes that GPT-6-astra has reconstructed. The idea is that if a scene program really captures the photograph, ordinary vision questions about the photograph can be answered by querying the program, with no vision model in the loop.

Among the evaluated agents, GPT-6-astra achieves the strongest overall results, scoring fifty-three point four percent indoors and thirty-nine point six percent outdoors. Even so, substantial gaps remain between delivering a valid scene artifact and faithfully recovering the scene's geometry and appearance.

Looking at how agents build their scenes, the analysis finds three recurring issues: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. In plain terms, agents start from a poor first draft, sometimes make the scene worse while trying to improve it, and cannot always tell when they have. Those findings motivate LEGO-Plugin, a training-free harness plugin for more controlled iterative construction. It improves all six evaluated models, with relative gains of up to sixty-two point seven percent in overall score. As for LEGO-World, its readouts show non-trivial performance on all three vision tasks, but fall well short of specialized vision models.

Producing a valid scene and recovering the right scene are separate achievements, and the benchmark is built to keep them apart. On that measure, current coding agents sit in between. The authors' summary is that program-constructed scenes from these agents are a promising but not yet sufficiently precise representation of natural images.