Training Object Permanence in World Models

Training Object Permanence in World Models. By Haotian Zhang and thirty others, from the University of Southern California, Carnegie Mellon University, and fourteen other institutions. Published on the twenty-third of September. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

Object permanence and solidity are hallmarks of human cognitive priors. Recent studies show video generation models, a paradigmatic class of current world models, beginning to display emerged reasoning abilities, which makes them candidates for building human-like physical intelligence. The paper asks two questions. Do video models already have object permanence? And if they do not, could they be trained into it with a dataset inspired by core cognition? Its response to the second is WROP, short for World Reasoning with Object Permanence: a data infrastructure built for that purpose.

At its centre are one hundred and fifty hand-designed tasks, each inspired by cognitive science, divided into six cognitive categories. Every task yields more than ten thousand samples. What gets released from this is a training corpus of one point five million samples, and an exam of three hundred questions.

The samples come from generators built in Blender. Each generator randomizes speed, lighting, camera angle and other nuisance parameters, while preserving the cognitive structure of its task. One task therefore appears under many surface conditions but always poses the same underlying problem, and that is how a single hand design becomes more than ten thousand samples.

The exam is aimed at video models, and the authors put fourteen of them through it. Three are reference-to-video models, seven are edit models, and four are continuation models. Among the continuation models is PWM-WROP, the authors' own sixteen billion parameter world model. In a blind pairwise Elo study it ranks first among continuation models and third overall. The only models above it are two reference-to-video models, and those two sit in a statistical tie.

The release reaches past the data and the exam. It includes the model answers, the scores, the weights, and PWM, the authors' native-PyTorch training stack on AWS Trainium2. Anyone picking this up gets the whole chain: the tasks, the generated samples, a model trained on them, and the record of how every evaluated model was graded.

Training Object Permanence in World Models. By Haotian Zhang and thirty others, from the University of Southern California, Carnegie Mellon University, and fourteen other institutions. Published on the twenty-third of September. This is the long version: the whole paper rather than the abstract.

Modern video generators produce footage that is photorealistic and coherent over time, and on that basis they are increasingly treated as world models: systems that simulate the world. Yet a characteristic failure persists. Objects vanish behind an occluder and re-emerge somewhere impossible, or pass straight through a solid barrier. These are failures of object permanence and object solidity, which infants show within the first half-year of life. The authors argue they sit structurally upstream: a model that lets objects interpenetrate cannot produce a valid collision or fall, and anything built on top inherits the error. So two questions. Do current video models respect these principles? And can they be trained in?

Two ideas come first. Classic infant experiments by Renée Baillargeon and colleagues used violation of expectation: show an impossible event and measure the response. Two of the six task families here are named after them. The difference is that a video model is never shown the impossible event. It must produce the plausible one.

The second idea is the evaluation contract. Every clip is split where the key physical event begins. The model receives the first half, the input, and must generate the second half, the target, which holds the event and its consequence. Existing benchmarks, the authors note, mostly score with vision-language models, which have documented deficits in exactly the core knowledge under test. So here the judges are people.

The benchmark is called WROP, for World Reasoning with Object Permanence. It is one hundred and fifty hand-designed Blender generators in six families. Three probe permanence: an object passing behind an occluder, a static group of objects hidden and then revealed by a moving screen, and an object inside a container that moves, rotates or swaps. Three probe solidity: an object meeting a barrier whose opening is too small or large enough, a support withdrawn from beneath an object, and collisions.

A detail the abstract leaves out is that there is no physics engine. Motion is authored as keyframe animation, every trajectory written analytically so that occlusion, contact and reappearance land on controlled frames. Physical plausibility is the scene author's responsibility, and scenes are revised whenever inspection finds objects passing through each other. Marked Boxes Swap shows the intent: two labelled boxes close over different objects, trade places and reopen, so final position alone cannot tell a model what is where.

Each generator yields ten thousand samples, one point five million in all. Every sample is a hundred and twenty frames split at frame sixty, with a prompt, a per-frame trajectory of object poses, and scene metadata. The exam takes two samples per generator, three hundred questions.

The trained model, PWM-WROP, is Cosmos3-Nano fine-tuned with only the training signal changed: same architecture, same tokenizer, no task labels. It sees one epoch of samples from an earlier render of the same generators, at a small frame size of three hundred and twenty pixels wide. The appendix documents PWM, its native PyTorch training stack for AWS Trainium2.

Fourteen models were given the input clip and prompt, never the target. They fall into three interface classes. True continuation models extend the clip. Reference-to-video models regenerate the whole event on their own timeline. Edit and transfer models repaint the source span frame by frame, so they cannot depict what happens after the occlusion boundary at all.

Twenty screened raters made blind pairwise choices, fitted with a Bradley–Terry model onto an Elo scale. Two reference-to-video systems, Wan 3.0 Prime and MiniMax H3, tie for first with identical records. PWM-WROP is third, the best true continuation model, about two hundred and twenty-four points ahead of the next one, Grok Imagine.

The comparison that tests the thesis most directly is by family. PWM-WROP ranks first on static occlusion and third on the other two permanence families, but fifth on obstruction and eighth on collision. The authors read this as fine-tuning helping occlusion tracking more than contact dynamics. The reference-to-video leaders show the mirror image, drawing much of their lead from solidity, where freedom to regenerate the scene helps. On automatic metrics at matched frame size, PWM-WROP has the best perceptual distance, zero point zero eight one against a next best of zero point one zero five, though the authors treat such metrics as secondary.

There is no limitations section. The caveats are scattered through the results, and they are real. The human study is thin: each of the hundred and twenty model pairs was judged on only four items, so nothing was rated on anything like the full exam. The solidity families rest on thirty-six to forty-four games each, and the authors warn that ranks there indicate a tendency, not a significant difference. PWM-WROP's output is also far smaller than its rivals', a confound for any metric computed at high resolution.

The source has seams too. Related Work calls the benchmark by what looks like an earlier name, a few leaderboard figures in the prose differ slightly from the table, and the appendix promises the status of the end-to-end fine-tune on the Trainium stack without ever stating it.

The idea to carry away is that physical common sense can be written as a curriculum. Take the paradigms developmental science used to probe infants, turn each into a generator that holds the physics fixed while the surface varies, and you get as much training data as you want for a single principle. One fine-tuned model reached the top of its class, strongest at keeping track of hidden things and weaker where things collide. Whether the data or the base model is doing the work is what this evidence leaves open.