HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

HARMONY. Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis. By Shufan Sun and four others, from the University of Pennsylvania. Published on the twenty-second of September. Its code repository is worked out: about ninety source files, with a readme and a licence.

Compositional 3D scene reconstruction has been approached from two directions, and each has a weakness. Agentic reasoning provides semantic understanding of spatial relationships, but lacks precise alignment with the input image. Visual geometry foundation models predict dense point maps from input images, but their reconstruction quality is limited. So recovering a complete 3D scene from a single monocular image, with accurate relationships between objects and high-fidelity reconstruction, remains a challenge. One approach knows what should go where but does not match the picture closely. The other matches the picture but does not produce a clean, complete scene.

HARMONY is a hierarchical chain-of-thought framework that uses both. Agentic reasoning supplies the understanding of what goes where, and visual geometry supplies the alignment with what the image actually shows. The chain of thought is hierarchical: the scene is built up in stages, from the frame of the room to the smallest objects, with reasoning at each stage.

Given an image of an indoor scene, HARMONY starts from an empty 3D floorplan. It first calibrates the camera against the reference image, to establish a spatial frame grounded in the scene's semantics. Everything that follows is placed within that frame. Then agentic VLM reasoning recovers the 3D room layout and an initial order in which to place objects.

Objects are placed hierarchically: first elements mounted on walls, then free-standing furniture, then decorations that depend on the furniture beneath them. Furniture is handled by depth-first traversal, so each placement is conditioned on structure already resolved, and a reflective feedback loop guards against errors accumulating. After the VLM places each object, point cloud estimates drive a geometry-based refinement, so that the rendered image aligns more closely with the input. The ordering is what makes the process work step by step. A decoration can only sit correctly on a table that is already in place, and a table can only be placed correctly once the walls are known. Each correction against the point cloud then keeps small misalignments from carrying forward into everything placed afterwards.

The result is 3D scenes that are semantically consistent and perceptually aligned with the reference image, extending single-image compositional reconstruction to complex indoor scenes. On synthetic and real-world images, HARMONY outperforms the reconstruction baselines evaluated. Qualitative comparisons with GPT-6 Astra suggest more faithful object arrangements and better preservation of scene detail. The combination of the two is the claim. Scenes come out both sensible, in the relationships between objects, and faithful, in how closely they match the photograph they came from.

The design divides the labour along the line where each approach was weak. Reasoning decides the order and the relationships, one object at a time, and geometry corrects each placement against the image before the next one is made.