Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

Puffin-World. Scaling a Unified Multimodal Model with Native 3D World States. By Kang Liao and nine others, from S-Lab at Nanyang Technological University, the University of Michigan, and two other institutions. Published on the third of September. No code repository is linked to it.

Systems that generate and reconstruct 3D worlds are usually assembled rather than trained. A generator here, a depth estimator there, a camera module bolted on, each one an external offline component doing its own job and handing results along. That arrangement makes it hard to keep a world physically consistent, because nothing in the pipeline holds physics, geometry and appearance in the same place at the same time.

Puffin-World is a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. The change is what the model treats as native. Rather than producing images and inferring the rest, it jointly models three world states directly: physics, meaning the gravity field and latitude; geometry, meaning depth; and appearance, meaning the image itself.

Those three states sit alongside a unified Omni-Camera representation, which is what lets one model support diverse tasks and flexible motions rather than one fixed viewpoint convention. On top of the states comes a strategy for propagating physical dynamics across future frames, so a world stays consistent as it runs forward rather than only at the moment it is generated. Grounding absolute camera properties in the real world is what makes generation physically consistent and visually stable. Appearance and geometry are then coupled inside a single generative process, synthesizing each future view and reconstructing its underlying geometry together rather than in sequence. That coupling is what enables interleaved closed-loop applications, including mimic and self-calibrated world exploration.

Scaling this to complex scenarios needed data built for it. Puffin-16M comprises fifteen million vision-language-camera triplets and one million trajectories featuring varied and challenging motions. The code, models and datasets have been released.

The claim being made is about architecture rather than results. Physics, geometry and appearance were treated as one modeling problem instead of three modules, and the closed-loop behaviour is what that buys.

Puffin-World. Scaling a Unified Multimodal Model with Native 3D World States. By Kang Liao and nine others, from S-Lab at Nanyang Technological University, the University of Michigan, and two other institutions. Published on the third of September. This is the long version: the whole paper rather than the abstract.

The stated goal is a model that can perceive, generate and reconstruct the world from arbitrary visual observations. The authors are precise about what that demands, and the precision is the interesting part. A capable world model should not merely generate plausible pixels. It should know where the camera is in the real world, reason about the geometry of the scene, and synthesize new viewpoints in a way that stays consistent with physical law. Which means the physical world cannot adequately be represented as a collection of two-dimensional images.

Progress so far has run along two separate tracks, each capturing part of the picture. Generative world models have advanced strikingly, but they predict at the appearance level: each frame is RGB content, with no explicit notion of the camera's physical orientation or the scene's underlying geometry. Unified multimodal models couple understanding and generation inside one network, but they do it for two-dimensional semantics. So the field still lacks a single model unifying the modalities and tasks that three-dimensional world modelling needs. The paper's diagnosis of why existing systems drift is the sharpest part of the setup. They rely on relative camera motion, which has no global physical reference frame, so the same relative trajectory can correspond to different absolute world orientations. That produces orientation drift, and it gets worse under challenging motion, over long horizons, and from a single view.

Puffin-World models three native world states jointly and end to end. Physics, which grounds an observation in the absolute physical world through gravity and orientation. Geometry, meaning depth and spatial structure. And appearance, the visual content we finally see. Around those sits the Omni-Camera representation, a dense action signal combining a gravity-anchored absolute field with a relative ray field, which is what lets one model handle both single-view text-to-image spatial simulation and cross-view image-to-3D generation without switching conventions. On top of that comes physics propagation: absolute spatial knowledge derived from perception is anchored, then propagated across the relative control signals of future frames, so generated appearance stays stable and gravity-consistent through complex motion. Appearance and geometry are then coupled inside a single generative process, synthesizing each future view while reconstructing its underlying geometry, rather than generating first and inferring structure afterwards.

On physical world perception the model reports state-of-the-art camera-to-world understanding, with the best median error and the best area under curve at five degrees across four public benchmarks. For free-viewpoint spatial simulation it produces camera-controllable images with a more faithful distribution than strong general-purpose generators. And for three-dimensional world modelling it does action-conditioned generation from text or an image while jointly reconstructing per-view geometry. The applications the authors highlight are the ones needing several of those at once, which is the actual argument for unification: mimic, and self-calibrated world exploration, where the model uses its own understanding of absolute camera physics to keep itself oriented as it explores. Code, models and datasets are released.

This paper has no limitations section, and its only sustained limitations discussion is aimed at prior work rather than at itself, which is a gap to name plainly. What can be established from the text is narrower than the claims might suggest. The perception results are camera-to-world understanding on four benchmarks; the generation results are reported as more faithful distributions than general-purpose generators rather than against a physics benchmark. Nothing here measures whether the generated dynamics obey physical law over time, which is a different question from whether the frame is correctly oriented, and it is the question a world model used for predicting consequences would eventually have to answer.

The claim to hold onto is architectural rather than numerical. Physics, geometry and appearance were treated as one modelling problem instead of three modules handing results along, and the closed-loop behaviour is what that buys. The part most likely to outlive the specific model is the diagnosis underneath it: that relative camera control with no absolute anchor lets orientation drift by construction, and that a gravity-aware representation is the thing that closes it.