Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang
Harbin Institute of Technology · Nanyang Technological University · Shandong University · Harbin Institute of Technology (Weihai) Qingdao Research Institute
Published September 10, 2026 · on the feed September 11, 2026 · ▲ 40 · arXiv 2609.11561
The abstract
Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.
The Brief
2:37A short podcast summary of the paper. Written and voiced by AI.
Transcript
Memory as Plans. World-Action Modeling with Memory-Grounded Planning. By Sizhe Zhao and seven others, from the Harbin Institute of Technology, Nanyang Technological University, and two other institutions. Published on the tenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Most robot policies are Markovian: they decide what to do from the current observation alone. But many complex real-world manipulation tasks are not like that. They need memory stretching far beyond what the robot can see right now. The memory mechanisms that exist tend to use language summaries, visual windows that keep growing, or a mix of the two. Summaries can lose fine-grained visual evidence. Growing windows force a trade-off between how much history is covered and how efficiently the policy runs. Either way, the robot either forgets detail it needs or pays more and more to remember it.
MaP-WAM, short for Memory-as-Plans, splits memory-dependent world-action modelling into two parts: planning grounded in memory, and execution conditioned on a plan. Long-term multimodal episodic context is used as evidence at planning time. The executor is not repeatedly conditioned on the full history. Memory informs the plan, and the plan is what the executor sees.
Memory is stored as records of completed segments, each containing language instructions and sparse visual context. That episodic memory is turned into compact plans, each made of the next segment-level language plan and the visual guidance that goes with it. A World-Action-Progress model, WAP, then carries out each plan over a duration that is not known in advance. At inference it predicts chunks of actions together with how far execution has progressed.
That predicted progress is calibrated by aligning the plan with what the robot observes, which allows segment transitions to adapt and the context to be updated in a closed loop. The executor's context length stays fixed, and structured attention allows key-value caching in both planning and execution. In plain terms, the robot works through a task segment by segment, knows how far through the current segment it is, and moves on when what it sees matches what the plan said should happen.
MaP-WAM reaches state-of-the-art performance on RMBench, with a success rate of eighty-three point three percent, and achieves seventy-eight percent success on tasks with a real robot. As the task history grows, the executor's inference latency stays approximately constant.
The name carries the idea. Memory is not handed to the executor as a growing record of everything that happened. It is turned into a plan: what to do next, and what it should look like. That keeps the part that acts small and fast, however long the task has run, while the part that remembers stays rich.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A robot that forgets what it did, or slows down remembering. Neither will do.
You are building a robot policy for tasks where what matters may no longer be visible: which block you already moved, whether a button was pressed.
The road ahead
- 1 · The problem ~7 min A robot often needs to remember things it can no longer see, but the usual ways to give it memory either lose detail or slow it down.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Use the long history only when planning the next segment, and let a fixed-context executor follow that compact plan.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min MaP-WAM loops through five steps: record, plan, act with progress, calibrate progress, then transition and update memory.
- 6 · The experiment ~6 min You’ll guess: As the task history grows longer, what happens to MaP-WAM's executor inference latency?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.