Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

Learning What to Recall. Adaptive Multi-Cue Episodic Memory for World Models. By Beomsu Kim and five others, from KAIST, Sony Group Corporation, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about one hundred and twenty source files, with a readme and a licence.

A world model predicts future observations from current experience and actions. Sometimes, though, the right prediction depends on something observed long ago. Episodic memory keeps past observations around for later recall, but as that memory piles up, a basic question appears. Which memories help with the prediction at hand, and which of the available retrieval cues should be trusted to find them? Fixed rules based on recency, on pose overlap, or on visual similarity can be unreliable across environments and across queries. The most recent frame is not always the useful one, a matching pose does not always mean matching content, and what looks similar may not be what the prediction needs. A rule that works in one place can mislead in another.

Future-Aware Recall, or FAR, learns how to recall instead of following a fixed rule. It combines two ingredients: supervision that looks at the future, and adaptive scoring across multiple cues. The guiding notion is predictive utility. A recalled memory is good if it makes the future that actually happened more likely. That shifts the question from which memory resembles the present to which memory helps predict what comes next.

During training, FAR measures that utility as the conditional log-likelihood of the realized future given the recalled context, approximated by the negative of a diffusion prediction loss. That signal trains a retriever. The retriever itself remains blind to the future at inference, so the future is used only to teach it, never to run it. Along the way it learns how relevant each cue is, and decides for each query which of the available cues to trust, whether time, pose, vision or audio, when choosing memories.

The evaluation spans three complementary settings, and the abstract reports three results rather than figures. First, FAR outperforms hand-designed recall even when both draw on the same retrieval cues, so the gain comes from how the cues are used, not from having more of them. Second, it adapts on its own which of the available cues it trusts. Third, it recalls the right history as the world around it changes.

Hindsight is the teacher here. At training time the model knows what came next, so it can grade each memory by how much it would have helped; at test time that knowledge is withdrawn and the retriever has to have learned the habit. The authors present FAR as a flexible and principled approach to reaching episodic memory in world models.