World-in-Context: Interactive Video World Modeling with In-Context Conditioning

World-in-Context. Interactive Video World Modeling with In-Context Conditioning. By Harold Haodong Chen and six others, from Alibaba Token Hub at Alibaba Group. Published on the twenty-third of October. Its code repository is a placeholder: a readme, and nothing to run yet.

An interactive video world model has to coordinate three kinds of control at once: the viewing perspective, what the character does, and how the camera moves. Existing methods handle these awkwardly. View modes are usually left implicit in text captions. Action tends to be conflated with camera motion. And the controls are injected through fragmented architectural pathways, so the dependencies between them are captured only indirectly.

World-in-Context replaces those pathways with In-Context Conditioning, or ICC. View, action and camera controls all become explicit tokens, placed directly in a shared self-attention space alongside everything else the model attends to. No specialised conditioning branch is needed. The formulation surfaces two observations. Explicit view tokens disentangle dynamics that would otherwise conflict, and they unlock view switching that emerges even from single-view data. And action and camera inputs turn out to play complementary roles: one governs local responsiveness, the other global stability.

Interaction also unfolds over time, in execution intervals, and continuity across those intervals needs its own treatment. For that the paper introduces Interaction-Transport RoPE, or IT-RoPE. It parameterises temporal attention with the control transitions accumulated so far, so the model's sense of time carries the history of what it was asked to do.

On WBench, the five billion parameter World-in-Context model reaches navigation performance on par with EchoWM at twenty-two billion parameters: eighty-one point five against eighty-one point six. For perspective switching it attains a ninety-three point eight percent success rate. Existing baselines at the five billion scale reach eighteen point eight percent. Controlled analyses add three more points: the advantages compound under richer mixtures of conditioning, data efficiency is favourable, and the approach applies naturally to causal rollouts.

The argument here is architectural economy. Put every control into the same attention space as explicit tokens, and the interactions between them no longer need to be engineered. The model can learn them, and in the case of view switching, a behaviour appears that the training data never directly showed.

World-in-Context. Interactive Video World Modeling with In-Context Conditioning. By Harold Haodong Chen and six others, from Alibaba Token Hub at Alibaba Group. Published on the twenty-third of October. This is the long version: the whole paper rather than the abstract.

An interactive video world model is a simulator you steer. You press keys or move a camera, and the model predicts what you would see next. Two things have to be controlled at once. One is how the world is viewed: from a free-floating camera, from a first-person perspective, or from behind a character in third person. The other is how your inputs move both the character and the camera. The authors name two failures in current systems. View mode is usually left implicit in a text caption, and free-camera roaming gets treated as if it were first-person walking, even though the two move differently. And most pipelines steer everything through a continuous camera trajectory alone, which in third person tangles together two separate degrees of freedom: where the character goes, and where the camera looks.

Picture how controls usually enter a video diffusion transformer. Each signal gets its own side channel. Camera trajectories modulate the network through AdaLN, discrete actions are added onto the visual tokens. The objection is that this parks controls in separate parameter spaces, so the dependencies between them are captured only indirectly, and coordinating the branches gets harder as controls multiply. Those dependencies are real: pressing forward means one thing in first person and another in third, and its meaning shifts again with where the camera currently points. The backbone throughout is Wan2.2-TI2V-5B, a flow-matching video generator, and every five-billion-parameter baseline in the main comparison is built on that same foundation.

World-in-Context, built on what the authors call In-Context Conditioning, removes the side channels. Every control becomes a token the same width as the backbone's hidden states, and those tokens are placed in front of the visual latents, so ordinary self-attention sees controls and video as one sequence. View tokens are learned categories, one per time step: free, first-person or third-person. With those in place, the text caption is restricted to the static world, things like weather, geometry and layout, and all motion is left to the control tokens. Action tokens split each input into slots: move and turn as discrete keys, plus continuous increments of translation and rotation, each slot with its own encoder and a zero-initialised residual branch carrying magnitude. Camera tokens encode each relative pose as a field of Plücker rays, combined with the global pose and the local rigid-body step.

Tokens say which control is active at each frame. They do not say what happened between two distant frames, and standard 3D-RoPE only encodes relative position in space and time. Interaction-Transport RoPE, or IT-RoPE, adds that missing history. Each control input, kept on separate paths for action and for camera, defines a small step transformation, and these are multiplied together along the sequence to give a transport from one frame to another. Because the steps need not commute, two trajectories that end in the same place by different routes produce different transports. That transport, minus the identity, adds a correction to the temporal attention score between two frames. Subtracting the identity means the correction vanishes when a frame meets itself, and over intervals where nothing moved. A learnable scale starts at zero, so training begins from the untouched backbone, and a gate conditioned on view mode and token type sets how much each path counts. The appendix shows the cost grows linearly with sequence length.

On WBench, the five-billion-parameter model averages eighty-one point five on navigation, against eighty-one point six for the far larger EchoWM. The fairer comparison is Zing-0.5, on the same backbone, at eighty-one point zero. World-in-Context wins on interaction, eighty-six against eighty-four point two, but Zing-0.5 scores higher on consistency and on physical plausibility. The real gap is perspective switching: ninety-three point eight percent success, against eighteen point eight for the best five-billion baseline.

That headline number has a history the abstract compresses. Trained only on single-view clips, simply changing the view token mid-rollout already switches perspective thirty-seven point five percent of the time, against six point two five for a baseline that names the view in text. Ninety-three point eight comes after a second stage that mixes five hundred switching clips, oversampled, into ordinary navigation data. The test that most directly probes the thesis is the controlled one, where additive injection and AdaLN are reimplemented on identical data and an identical conditioning schema. With camera control alone, the three strategies tie. Adding action opens a small margin for in-context conditioning, and adding view tokens widens it to about one point on navigation. Separately, keeping only the endpoints of the transport instead of the intermediate steps drops navigation from eighty-one point two to eighty point five, the same as plain 3D-RoPE.

There is no limitations section. A closing note on positioning says the work is about formulation rather than a complete system, and that data-scale comparisons are left out of the main table because other methods rarely disclose their training data. The full text adds caveats the abstract does not. Action labels are mostly not recorded but inverted from camera trajectories estimated with ViPE, with first-person head-bobbing filtered out first. Joint training means a mixture of action-only, camera-only, joint and unconditioned samples, not both signals on every clip, and at inference free and first-person modes default to action alone. The controlled analyses use the model before the switching stage. Every result is a single number, with no error bars or repeated runs.

The idea to carry away is that controls can be context rather than plumbing. Put them in the sequence the video lives in, and the model can learn how they bear on each other. Then give attention a memory of the route between two frames, not just the distance. The evidence says the benefit is small for one signal and grows as signals pile up, and that an explicit view token turns perspective switching from something these models rarely manage into something a few hundred examples can teach.