Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies

Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies. By Chubin Zhang and one other, from Nanyang Technological University. Published on the tenth of November.

Memory-augmented vision-language-action policies are meant to remember what happened. Many of them, though, fill memory by time. They sample frames uniformly across the episode and stamp each frame with its time step. Time passes even when nothing happens, so a memory built this way may not be able to tell waiting from working. The claim is that it cannot, and that the policy leans on a temporal shortcut: elapsed time standing in for progress. The title puts it as a question, whether a robot's memory is just a clock, and for the memory designs examined the answer is closer to yes than the designs intend.

The test uses the counting tasks of RoboMME. Partway through, the robot briefly pauses, or fidgets its wrist. A null event like that changes nothing the task depends on. Yet a released memory VLA counts the interval as work done, and stops early. The authors name this failure phantom progress. On one task it cuts success from forty-seven out of fifty down to sixteen. On the other, from forty-four down to ten. In both cases roughly two thirds or more of the successes disappear, and nothing about the task itself was made harder.

Two controls point at the cause. When the same frames are withheld from memory, success returns, so the problem lies in memory rather than in the interruption itself. And rescaling the time stamps alone is enough to steer the policy: it stops early when the clock runs fast, and repeats when the clock runs slow. The reason it can lean on the clock is in the training data. Demonstrations proceed at a steady pace, so elapsed time predicts progress.

Removing the time encoding does not cure the failure, which is the surprising part. The frames of a null event still occupy slots of the time-sampled memory, so the clock is built into what gets remembered, not only into the stamps. A pause that lasts long enough fills slots that would otherwise hold moments of real work, and the memory comes out looking fuller of progress than the episode was.

The evidence comes from the counting tasks of RoboMME and from its released memory designs. Across those designs, vulnerability follows how memory is indexed. Two fixes suggested by the diagnosis were tried, and they did not fare equally. Training with injected pauses does not remove the failure, and it impairs counting. Indexing memory by events recovers half or more of the loss, without retraining. Neither fix is reported as closing the gap entirely. The contrast between the two is itself informative. Teaching the policy to tolerate pauses leaves the time-based memory in place, while changing what the memory is indexed by goes after the shortcut directly.

Think of a robot asked to count, which watches itself stand still and decides the count went up. That is the failure in miniature, and the remedy follows from its cause. A memory meant to remember what happened should count events rather than clock ticks.