Execution Before Memory: Diagnosing Memory Failures in Video World Models

Execution Before Memory. Diagnosing Memory Failures in Video World Models. By Weiyang Jin and four others, from the University of Hong Kong, the University of Michigan, Ann Arbor, and one other institution. Published on the sixteenth of October. Its code repository is public and real: sixteen source files.

A memory test for a video world model ought to answer three separate questions. Was the scene set? Was the requested change actually carried out? And does the last frame show the result? Current tests fold all three into a single number. That makes a low score easy to misread as forgetting, when it might mean something else entirely. Consider the case where the change was never carried out. Then the hidden state transition could never appear in the first place, and there was nothing for the model to remember. In that case the test was not really testing memory at all. It was testing whether the instruction got executed.

MIST is an evaluation built to pull those questions apart. It separates the change itself from the memory that change should leave behind. The aim is attribution: when a model fails, the failure should be assigned to the stage where it occurred, instead of being booked as forgetting by default.

The structure is staged. First, the model sets the scene while the change stays hidden from view. Only later is it asked to show the change, and any leakage before that point is kept under control, so the reveal cannot be given away early. Because the steps are separated in time, a low result can be traced to the step where it actually happened: setting the scene, carrying out the change, or showing what was remembered.

That separation locates the bottleneck, and it is not memory. The model sets the scene, but it does not carry out the instructed change. Nor is this specific to MIST: the same gap appears in general memory evaluations too. Building on that diagnosis, the authors propose M3-Agent, short for Make the Missing Move. It recursively revises the details of the input instruction until the model successfully carries out what was asked. Rather than changing the model, it changes how the request is put to the model.

The effect is large. Relative to the baseline, M3-Agent more than triples the MIST reveal score. On other benchmarks, WRBench's score T rises by twenty-six point one percent, and MBench-T's score U rises by fifty-two point seven percent. The model stays frozen throughout. What changes is only how clearly the move is stated.

A low memory score, on this account, may be an execution score in disguise. M3-Agent meets the problem from the execution side, showing what a frozen model can do once the move is spelled out. In doing so it sets a stricter baseline, one that any future claim about memory in video world models now has to clear.