WorldSculpt: Generating Compositional Worlds from Grounded Videos

WorldSculpt. Generating Compositional Worlds from Grounded Videos. By Muyao Niu and eleven others, from Alaya Lab and the University of Tokyo. Published on the fourth of September. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.

The problem is generating a compositional 3D representation of a cluttered scene holding hundreds of objects. Compositional means something specific here: the scene as a collection of individual object meshes placed in a shared world frame, which is what gaming, AR and VR, simulation and robotics actually require. Dense clutter makes it hard. Objects heavily occlude one another, and any single view reveals only a fraction of each object's geometry. Existing work splits two ways and both fall short. Geometry-based approaches reconstruct the scene as a single representation and leave incomplete geometry wherever something was occluded. Compositional methods with generative priors handle occlusion, but have been largely limited to relatively simple scenes.

The claim made here is that complex scenes with hundreds of objects can be generated compositionally after all, by adapting a strong single-object 3D generative prior to multi-view observations. The prior already knows what a whole object looks like, including the parts no camera saw. What it has lacked is a way of being told which object, observed from where. That is a strong thing to lean on, and it is leaned on deliberately, in place of adding scene-level supervision.

That paradigm is instantiated with Pixal3D, extended with a multi-view conditioning pathway which grounds object generation in multiple posed observations rather than in one view. The detail to notice is where the training happened. The model is finetuned entirely on single objects in canonical space, with no scene-level training at all, and it still generalizes to large scenes under severe occlusion. The absence of scene-level training is the load-bearing part, because it is what demonstrates both feasibility and scalability: nothing has to be annotated at the scale of a whole scene.

To measure that, the authors introduce UE-MeshyScene, a photorealistic benchmark of densely cluttered scenes with hundreds of objects, carrying per-object annotations and ground-truth meshes. Across three evaluations, single-object, controlled multi-object, and UE-MeshyScene itself, the method consistently outperforms prior approaches. The pattern inside those results is the informative part: the gains grow larger as scene complexity and occlusion increase. A method pulling ahead precisely where the problem is hardest is behaving the way its own explanation predicts.

Applicability beyond the benchmark is shown by conversion. Generated 3DGS worlds, Marble and HY-World 2.0 among them, are turned into compositional mesh scenes. The through-line of the design is that scene-level understanding was never the missing piece. A prior trained on single objects, given enough posed views to know which object it is looking at, produces the scene without ever having been shown one.

WorldSculpt. Generating Compositional Worlds from Grounded Videos. By Muyao Niu and eleven others, from Alaya Lab and the University of Tokyo. Published on the fourth of September. This is the long version: the whole paper rather than the abstract.

Generative world models can now produce persistent, explorable 3D environments from a single image or a short prompt, with plausible geometry well beyond what was directly observed. But the world they produce is a unified representation: one fused mesh, or a set of Gaussians. A chair in that world is not a chair. It is a region of surface. You cannot select it, move it, or hand it to a physics simulator, because it was never represented as a separate entity. The paper's phrase for what downstream applications will not accept is exact: they do not consume an undifferentiated soup of surface geometry.

A faithful result has to satisfy three requirements at once. Align the visible evidence across images. Recover plausible geometry for the regions no view showed. And preserve each object as an independently usable asset. Existing families each manage some of that. Geometry-centric reconstruction is accurate where it can see and leaves holes where it cannot, in one fused representation. Scene-level generative models fill the unobserved regions and produce something monolithic. Native 3D generative priors model complete individual objects beautifully, but are built for a single pose-free image of one object, with no way to condition on a consistent set of views or to generate in a known scene coordinate frame. And compositional methods that extend object priors to multiple objects have been demonstrated on tabletops and sparse furniture arrangements.

The approach keeps a strong single-object prior and extends it rather than replacing it. Inputs are images with instance-level masks and three-dimensional bounding boxes. For each object, an anchor-aligned canonical frame is built from its observations across the scene, and per-view visual features are lifted into the corresponding canonical voxel volume. Features from different views are then fused by an aggregator that is permutation-invariant and accepts a variable number of inputs, which matters because how many views see a given object is not something you control in a cluttered scene. The fused condition enters the frozen prior through zero-initialised projection layers, with low-rank adapters letting the pretrained network exploit the new evidence. Each object is generated in its own canonical frame and placed by a canonical-to-world transformation, with no cross-object fusion and no joint shape optimisation.

Measuring any of this needed a benchmark that did not exist, so the authors built one in a game engine: large indoor and outdoor environments holding up to several hundred assets in natural configurations with substantial mutual occlusion, shipped with camera parameters, bounding boxes, and ground-truth meshes per object. That combination is the point. Real captures of cluttered scenes are realistic but come with no exact geometry to score against; rendered scenes supply both. Evaluation then runs across a progression of difficulty, from single-object stress tests, through real-captured and synthetic multi-object scenes, to the heavily cluttered benchmark, and the gains grow as scenes become more crowded and occluded.

The framing carries real assumptions. Instance masks and three-dimensional bounding boxes are inputs, not outputs, so the decomposition into objects is given to the system rather than discovered by it. Objects are generated independently and placed without joint optimisation, which is what makes it scale and also means nothing enforces consistency between neighbours, such as two objects not intersecting. The headline benchmark is rendered rather than captured, buying exact ground truth at some cost in realism. And the reported advantage of the learned fusion is not uniform: at some view counts plain averaging is marginally better, which the authors report rather than omit.

The demonstration that closes the paper is the one that shows what this is for. Take a world generated by an existing scene-level model, render a sequence of posed views of it, and run the pipeline over those renders. What comes back is the same world as a collection of individual meshes in a shared frame. The monolithic output of a generative world model becomes compositional, after the fact, by being looked at from several angles and rebuilt one object at a time.