Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

Right Screen, Wrong Transition. World Models as Verifiers for GUI Agents. By Jiaming Zhang and five others, from Nanyang Technological University, the Institute of Science Tokyo, and one other institution. Published on the eighth of October. Its code repository is public and real: about thirty source files.

Picture a login screen. If it appears after a tap on Sign in, that is exactly what should happen. If the same screen appears after a tap on View order, it is an attack. The screen is identical in both cases. What differs is how the agent got there. For GUI agents, the authors argue, safety is a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated simply by reusing a legitimate one.

Judging a transition needs an expectation of what should have followed the action. Existing GUI world models can supply that expectation, but they output it as text, code or images. Checking such an output against the screen that actually appeared then requires a second model to judge the two against each other. The expectation exists, but it is written in a different language from the observation, and translating between them becomes a job of its own.

The proposal is that a world model meant for verification should predict in the space where observations are already encoded. The authors present LGWM, a decoder-free, action-conditioned world model. Given a screen and an action, it predicts the representation of the next screen directly, with no image or text to generate. Verification then reduces to a vector comparison between the predicted representation and the observed one. There is no second model in the loop to arbitrate. And the same signal does a second job: beyond showing that a transition broke expectation, it also indicates whether that mismatch is harmful.

LGWM is trained without semantic annotation on one point eight five million real GUI transitions. No one has to label what a screen means; the training material is simply real screens and the actions that led between them. To evaluate it, the authors build RSWT-BENCH, a diagnostic in which each credential screen appears twice: once under a legitimate transition and once under a hijacked one. That construction is the point. A detector that sees only the screen is at chance by design, so any success has to come from reasoning about the transition.

The training-free score reaches an AUC of zero point nine eight seven, at seventeen milliseconds per decision. That is on par with the strongest closed-source VLMs, and about ten AUC points above generative GUI world models, at more than three orders of magnitude lower latency.

A second test asks whether a violation is harmful or benign, which is a harder question than whether a violation happened at all. Here the residual direction reaches an AUC of zero point nine five three at separating the two, while prompted VLMs are near chance. Further analyses indicate that the prediction is a usable future state, not merely an anomaly score.

World models have mostly served as simulators or as planners. This work points to a third role, verification, and argues that for that role predicting in representation space is the natural design. The login screen example sums it up: nothing about the screen is wrong, only its arrival, and a model that expected something else can tell the difference with one comparison of vectors.