Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
Yansong Shi, Jiange Yang, Xijie Yang, Shaowei Zhang, Yuhan Zhu, Tao Lu, Limin Wang
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Zhejiang University · Shanghai Jiaotong University · Nanjing University
Published September 30, 2026 · on the feed October 2, 2026 · ▲ 16 · arXiv 2609.38886
The abstract
Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce HIDE, a benchmark for evaluating manipulation memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose SEEK, a framework combining three complementary memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching memory design to task-specific information requirements.
The Brief
2:22A short podcast summary of the paper. Written and voiced by AI.
Transcript
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation. By Yansong Shi and six others, from the University of Science and Technology of China, the Shanghai Artificial Intelligence Laboratory, and three other institutions. Published on the thirtieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Robot manipulation policies now handle increasingly diverse tasks and generalize across environments. Yet reliable execution often hinges on a task state the robot cannot see. How many times has something already been done? What was where, a moment ago? How far along is the job? None of that can be read off the current observation, which makes interaction history essential, and it is exactly what an evaluation built on single observations would never probe.
HIDE is a benchmark for manipulation memory under partial observability. It measures three kinds of remembering. Repetition counting asks a robot to keep track of how often something has happened. Historical-state recall asks it to remember how things stood before. Execution-progress tracking asks it to know where it is in the task. Each is a piece of information the current camera frame cannot supply, so a policy can only succeed if it carries the information forward from earlier.
There are fifteen tasks. Initial configurations are randomized, and each task contains decision points where similar observations demand different actions depending on what happened earlier. A policy that looks only at the present cannot tell those moments apart. Alongside the benchmark, the authors propose SEEK, a framework that combines three complementary memory mechanisms to retain historical evidence and track execution state.
Existing policies show substantial limitations on HIDE. Adding memory improves task success, both in simulation and in real-world experiments. The individual mechanisms do not behave uniformly, though: each helps some tasks and can hurt others. Combined, they reach the highest average success rate on HIDE among the configurations evaluated. So the gains from memory are real, but they depend on which memory meets which task, and only the combination comes out ahead on average.
Mixed results from single mechanisms carry the lesson. No one kind of memory serves every task, and the authors conclude that a policy should keep internal representations of hidden task state, with the memory design matched to what each task actually needs to know.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The scene looks exactly the same as a moment ago. So how does the robot know what to do next?
You are asked to make a robot press a button exactly twice. After the first press, the scene looks just as it did before. The camera cannot tell you how many presses are done.
The road ahead
- 1 · The problem ~6 min Some robot tasks need facts the camera cannot show, yet most policies mainly look at the current view and most tests don't isolate that need.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Name the hidden facts robots need, build tests whose look-alike moments demand them, and give the policy memory suited to each kind.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build history-dependent tasks, randomize their setups, add memory that keeps past evidence, combine the three memories, and test in simulation and on a real robot.
- 6 · The experiment ~6 min You’ll guess: When each of SEEK's memory mechanisms was used on its own across the benchmark's tasks, what happened?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.