Reading, Writing, Sharing, and Interacting with the World through Video

Reading, Writing, Sharing, and Interacting with the World through Video. By Shengqiong Wu and seven others, from the University of Oxford, the National University of Singapore, and two other institutions. Published on the twenty-second of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Two lines of work have been moving fast. Video foundation models have advanced at understanding and generating visual sequences. Video world models aim at something underneath the pixels: the dynamics of the world the video depicts. Their goals differ, but both are about the same relationship, between what a video shows and a world that is changing. This survey takes world modeling as the lens that unifies them.

The starting point is a way of seeing video. An observed video is treated as a projection of a hidden world that unfolds in time, and the projection is partial, lossy, and ordered in time. On that view, the literature is organised around four operations that complement one another.

Reading infers properties of the world from observed video. Writing goes the other way, synthesizing video from world conditions that are either specified or learned. Sharing brings understanding and generation together through modeling mechanisms they have in common. Interacting predicts and updates the state of the world as actions are taken and new observations arrive.

One distinction falls straight out of this framing. Temporal coherence in a sequence, whether observed or generated, is not the same thing as consistency in the underlying state of the world. A video can hang together frame to frame without the world behind it holding together.

Looking across tasks, representations, models, datasets and benchmarks, the authors trace a shift in emphasis. The earlier focus was on recognition at the level of a clip and on perceptual fidelity. The emphasis is moving toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and in the decisions made downstream. The survey closes by distilling the open challenges that follow from that shift.

Understanding video and generating it start from different objectives. Through this lens they become two directions of the same exchange with a hidden world, one reading it and one writing it, joined by sharing and extended by acting.

Reading, Writing, Sharing, and Interacting with the World through Video. By Shengqiong Wu and seven others, from the University of Oxford, the National University of Singapore, and two other institutions. Published on the twenty-second of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Two lines of work have been advancing side by side. Video foundation models have improved rapidly at understanding and generating visual sequences. Video world models try to capture the underlying dynamics of the world those sequences depict. Their objectives differ, but both are about the same relationship: between what a video shows and a world that keeps evolving behind it. This survey uses world modeling as the lens that unifies them.

The starting view is that an observed video is a partial, lossy, temporally ordered projection of a hidden world that changes over time. From there the literature is organized around four complementary operations. Reading infers properties of the world from observed video. Writing synthesizes video from world conditions, whether specified or learned.

Sharing brings understanding and generation together through common modeling mechanisms. Interacting predicts and updates the state of the world under actions and evolving observations. The survey traces these across tasks, representations, models, datasets, and benchmarks. The four are described as complementary, not competing: each picks out a different thing a model can do with the relationship between a video and the world behind it.

One distinction runs through the whole framing. Temporal coherence in a sequence, whether observed or generated, is not the same thing as consistency in the underlying world state. A clip can flow smoothly and still describe a world that does not hold together. The first is a property of the pixels on screen; the second is a property of the world the pixels are supposed to be showing.

Following that distinction across the literature, the authors trace a shift in emphasis. The field is moving away from recognition at the level of single clips, and from perceptual fidelity, toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and in downstream decisions.

The survey ends by distilling the open challenges that follow from that shift. Its four verbs give a compact way to place any piece of video work: is it reading the world, writing it, sharing one model between the two, or acting inside it.

Reading, Writing, Sharing, and Interacting with the World through Video. By Shengqiong Wu and seven others, from the University of Oxford, the National University of Singapore, and two other institutions. Published on the twenty-second of October. This is the long version: the whole paper rather than the abstract.

Video models have split into families that rarely share a vocabulary. Video language models answer questions about clips. Generators such as Sora, Wan and HunyuanVideo synthesize them. Unified models try to do both in one backbone. World models such as Genie and V-JEPA 2 try to predict how an environment changes when something acts on it. This survey, from Oxford, Merced and two other institutions, argues that all of them face one question: what relationship does a model learn between an observed video and the evolving world that produced it? It bites because a convincing clip is easy to mistake for an understanding of the world behind it.

Three objects are kept apart. The world is a hidden process of states: entities, their relations, and physical variables, evolving under optional actions. The video is a trace of that world, and a poor one in three ways. It is partial, because the camera sees only part of the state. It is lossy, because resolution, compression and editing throw information away, so different worlds can produce identical footage. And it is temporally ordered, so frame order itself carries evidence about motion. The third object is the represented state, whatever the model builds internally, which in general is not the true state of the world.

From that comes the distinction the rest of the survey leans on. A trace is temporally coherent if neighbouring frames agree locally: no flicker, no implausible jumps. It is world-consistent if at least one admissible world trajectory, respecting identity, layout, physics and causality, could have produced it. Neither implies the other. An object that changes identity after leaving the frame gives perfectly smooth frames that no world explains. A hard cut breaks smoothness while remaining entirely consistent. Coherence belongs to the trace. Consistency belongs to the world the trace implies.

Four operations organize the literature. Reading infers properties of the world from video: semantic, spatial, temporal, physical and causal. Writing samples a video from a condition, such as a prompt, an image or a camera path, which specifies the world only in part and leaves the generator to resolve the rest. Sharing couples a reader and a writer through at least one common component: parameters, tokens, an interface, or a training objective. Interacting rolls a transition model forward. The next state is predicted from the history, the actions and the conditions, and decoded to frames only when visible output is needed. One system can do several at once: an interactive world model reads its history, writes the next frame and updates its state in one backbone.

World models then get a two by three grid. One axis is prediction space. Observation-space models predict frames, video tokens, or generative latents that are decoded to video. Latent-state models predict video-derived features that never need rendering, as DINO-WM and the action-conditioned V-JEPA 2 do. The other axis is the transition interface. Passive models predict from observations alone. Action-conditioned models receive a control sequence fixed before rollout. Interactive models accept new controls during rollout, so each input changes what follows, as in GameNGen and Genie. A cell describes how a model is used, not what it is: the same latent predictor can score fixed candidate actions or sit inside a planner that acts, observes and replans.

Rollout exposes what a single clip hides. Under teacher forcing a model trains on clean history but deploys on its own outputs, so a small error in geometry becomes the next input. Diffusion Forcing answers by giving sequence elements independent noise levels, and Self-Forcing trains on the model's own rollouts. Memory raises a subtler problem. Most memories store what was visible, while a predictive state must also track what changed off-screen. A hidden ice cube keeps its identity as it melts, and replaying the stored observation returns a stale cube.

A survey's evidence is its synthesis, and its most useful piece is an evidence ladder with four levels for each operation: plausible output, faithfulness to the evidence or conditions, a consistent world state, and decision utility. Stronger claims need higher rungs. Applied to the field, the ladder shows where support thins out. For unified models, coverage of each direction separately is mature, integrated behaviour is emerging, and measured synergy is mostly directional: video evidence of transfer so far runs one way at a time. For world models, WorldArena finds that perceptual quality and downstream utility need not rank models the same way. Benchmarks can be fooled too: a static cache passes revisit tests even when the world should have changed, which is why MemoBench requires an object that changed out of view to return in its updated state.

There is no limitations section. The survey turns its critical apparatus outward, onto the field, and does not apply it to itself. Some boundaries are stated as scope: it excludes models whose main persistent state is explicit three-dimensional structure, such as neural fields or Gaussians, and purely symbolic or text-based world models. What a listener should hold in mind is that the framework is a lens, not a result. The definitions are conceptual, and the seven open challenges, from long-horizon consistency to stochastic worlds and rare events, are mostly calls for protocols that do not yet exist.

The idea to carry away is the survey's final insight: observational scale shows what co-occurs, and interventions show what changes what. Passive video, however much of it, leaves hidden state, physical causes, action meaning and outcome probabilities underdetermined. The common test it proposes is simple to state. Hold the initial state fixed, change one action or physical parameter, and check that the affected outcome changes while everything unrelated stays put. A ball released under two gravity settings should follow two trajectories against the same background. Until models are judged that way, a convincing video is evidence of rendering, not of a world.