When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?. By Anton Hristov and two others, from the Technology School Electronic Systems and Baba Tonka High School of Mathematics. Published on the first of November.

Prediction errors from video world models are often treated as a measure of physical surprise. If the model is badly wrong about what happens next, the reasoning goes, the event must have broken physics. This work argues that the inference does not hold reliably. A prediction error can also reflect appearance, static content, motion, or simply ordinary prediction difficulty. Its conclusion: a model can represent physical information, and can respond to incorrect actions, without its native prediction error reliably measuring physical surprise.

To pull those causes apart, the authors build an evaluation framework around three questions. What do frozen representations contain? How do they organize appearance differences against physical ones? And what actually changes a model's own prediction error? The first two are studied on IntPhys2, a benchmark of physically possible and impossible events, comparing representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance stays easy to decode, while V-JEPA2 exposes modest information about whether an event is possible or impossible. In the latent geometry, deeper layers show three trends. Appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases.

The third question is where the trouble shows. Across action-conditioned models, incorrect actions do raise prediction error, but appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout a clip can lower it. Using Kubric, a simulation and rendering framework, impossible events can produce lower error than possible ones. And a fixed error threshold incorrectly flags many possible events as impossible.

The claim is a careful one, and its limits matter. It does not say these models lack physical knowledge: the representations do carry some, and the models do respond to wrong actions. What it questions is one specific reading, that the native prediction error on its own is a dependable surprise signal. The evidence comes from the settings named: IntPhys2, the Physion++ clips, Kubric scenes, and the action-conditioned models tested.

The lesson concerns measurement more than models. High error can come from a frozen frame, low error from an impossible event, and a single threshold mislabels plausible scenes. Before a prediction error is read as surprise, the other things it could be tracking have to be separated out, and this framework is built to do that separation.