Memorizon: Training World Models Beyond Their Context Window

Memorizon. Training World Models Beyond Their Context Window. By Tingting Liao and three others, from MBZUAI. Published on the thirtieth of September. Its code repository is public and real: sixteen source files.

A streaming world model should render a place consistently each time it comes back to it. Walk away from a room and return, and the room should look the way it did. Supervising that directly means training on samples that contain both visits, and the two visits are often minutes apart. Dense attention over the full span between them incurs quadratic cost, so supervising long spans becomes expensive very quickly. The length needed for supervision and the length paid for in attention have been tied together, and the longer the revisit, the worse the bill.

Memorizon breaks that coupling. Its premise is that long spans are needed for supervision, but not for attention. The two visits can share a forward pass without including every frame that came between them. So a training sample may cover a span of any length, but it is scored only on its last k chunks, and only those chunks need to see the earlier visit.

Rather than tokenizing the whole history before those chunks, each scored chunk retrieves its own top K latents, chosen by camera co-visibility. In other words, it pulls back the earlier latents whose camera views overlap its own. The union of those requests forms a shared bank. The bank is bounded by k times K, so the sequence stays bounded however long the span grows. At the shortest span, the recipe is exactly conventional training, which means the method extends ordinary training rather than replacing it.

Cost behaves as designed. Adding the bank raises the cost of a training step once. Beyond that, a longer span costs little: going from one hundred seconds to four hundred seconds adds twelve percent to the step time. Against a sliding-window baseline, retrieval raises revisit consistency on every split. A span long enough to reach the first visit of each return adds a further twenty-four to thirty percent, at some cost in image quality. Past that span, extra length no longer helps.

A control checks that the bank is really being used, and is not just extra compute. Filling it from a different episode lowers revisit correlation by eighty-three percent. If the retrieved memories come from the wrong place, the consistency falls apart, so the model does rely on what it retrieves.

Two lengths that are usually one have been pulled apart. How far back the supervision reaches can stretch to minutes, while what the model attends to in a single step stays fixed in size. That separation is how training reaches beyond the context window named in the title, and the point where extra span stops paying off marks how far it is useful to reach.