World Models with Configurable Perception

World Models with Configurable Perception. By Yijun Shen and four others, from East China Normal University and the Hong Kong University of Science and Technology. Published on the twentieth of October. Its code repository is worked out: about one hundred and seventy source files, with a readme.

Real-world observations are noisy, and much of what they contain has nothing to do with the task at hand. So world modeling and planning need to abstract an observation into a state representation while filtering the irrelevant parts away. The encoders in existing world models are typically non-adaptive, and that is the problem. A fixed encoder yields state representations that are easily distracted by task-irrelevant information, and a model built on them cannot filter its observations selectively to suit one task demand rather than another.

ConfigWM makes perception configurable, through task-conditioned state extraction. The task arrives as a language description, and that description is used as a configuration signal. It steers how visual embeddings get mapped into a state space specific to that task, keeping what is relevant to the task at hand while suppressing variations that are not. The same scene, described with a different task, can be abstracted differently, because the description is an input to the mapping rather than something applied after it.

Dynamics are then learned inside that task-specific state space, conditioned on actions, and planning runs there. The order matters: the task shapes the state before any dynamics are fitted to it, so what the planner reasons over is already the filtered view rather than the raw observation.

The evaluation turns distraction up by degrees, through environments with increasing levels of it. Existing methods show substantial declines in planning success rate as the environments get noisier. ConfigWM stays comparatively stable, and across diverse environments it outperforms the strongest competing method by twenty-two point three percent on average.

Two further measurements speak to what the states contain. Physical-quantity probing improves by zero point three six five in Pearson correlation, and distracted-to-clean retrieval improves by seventy-five point five percent. The authors read both as indicating that ConfigWM learns states that hold on to task-relevant information while staying robust to distractors. One of those checks asks whether physical quantities survive the abstraction. The other asks whether a cluttered view and a clean one land in the same place.

A fixed encoder has to guess once, in advance, what will matter. ConfigWM asks instead, and the answer arrives in the same form the task is given in: a sentence. Perception becomes something the task configures, which is what the paper offers as support for reliable world modeling and planning in noisy worlds.