SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

SimpleMemVLA. A Simple but Effective Native-Video Memory for Vision-Language-Action Models. By Cheng Yin and ten others, from the Huazhong University of Science and Technology, the Renmin University of China, and five other institutions. Published on the second of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Long-horizon manipulation is partially observable. The information a robot needs for its next action may only have appeared in what it saw minutes earlier. Existing memory mechanisms, whether retrieval banks, learned compressors or recurrent states, all share one limitation: they must decide what to keep from the past before knowing what a future decision will require. That design grew from an assumption that minute-scale history is too large to process directly. The authors argue that modern VLM backbones no longer make that assumption true.

SimpleMemVLA is a VLA with no dedicated memory module at all. It keeps the sampled history intact and hands it to the backbone as video, with timestamps, in the very format the backbone was pretrained to process. Nothing is thrown away in advance. The choice of what to keep, which older mechanisms had to make ahead of time, is no longer made before the decision at all, because the whole sampled history is still there when the decision arrives. The backbone is used for the kind of input it already knows how to read.

One narrow channel carries history to action. The model generates a sub-task, and the hidden states of that generated sub-task are the only route from the history to a standard flow-matching action head.

Latency is handled by noticing that consecutive decisions share most of their history. The shared prefix is prefilled while the current action is still executing, which keeps latency close to that of a VLA working from a single frame.

SimpleMemVLA sets a new state of the art on four memory benchmarks, and does so without cost on general-purpose control. With the backbone and training setup held fixed, it outperforms retrieval, compression and recurrent-state mechanisms by a wide margin. Causal interventions confirm that the policy genuinely reads its history, rather than succeeding some other way. Code is available.

The paper's move is subtraction, and the name says as much. Instead of building a better memory, it removes the memory module and trusts a backbone already trained on video to read the past directly. The one piece of engineering that remains, reusing a shared prefix across decisions, is there to make that simplicity affordable in time. The result is a memory that never had to guess the future, because it kept its sampled past intact, and a policy the interventions show is actually using what it keeps.