From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

From VLM Planners to Robotic Agents. Self-Evolving Harnesses for History-Dependent Manipulation. By Tingwen Du and eleven others, from the University of Science and Technology of China, the Shanghai Artificial Intelligence Laboratory, and three other institutions. Published on the twenty-first of October. Its code repository is worked out: about one hundred and sixty source files, with a readme and a licence.

Vision-language-action models do well across embodied manipulation tasks by bringing visual understanding, language conditioning, and action generation into one model. Long-horizon tasks add a difficulty: they can depend on history. A good decision may require remembering how far the task has got and what earlier interactions produced, while partial observability and execution errors pile up over time. Existing work has started on this, by giving foundation-model policies memory, and by separating high-level planning from low-level execution in hierarchical architectures.

The approach here works on the high level. The vision-language model that plans is organised as a robotic agent, through a harness that can be edited. The harness maintains the task context, and the planning improves through execution experience.

The instance is called SERA, the Self-Evolving Robotic Agent. Low-level control comes from a fixed VLA executor, which does not change. Above it, the harness keeps and updates the task context, using observations and execution feedback to guide the next round of planning. Self-evolution is the second mechanism. Environment rollouts are used to propose revisions to the harness and to evaluate them. The configuration that gets selected is then fixed before any held-out testing begins.

That gives the harness two timescales. Within a task, it tracks context and feeds what happened back into the plan, which is how it handles progress and past outcomes that the current observation alone cannot show. Across rollouts, the harness itself is revised and the best revision is kept. The ordering matters for reading the results: selection finishes before testing starts, so the held-out score does not choose its own configuration.

The benchmark is RoboMME. With autonomous selection, SERA reaches fifty-one point three eight percent overall success. Without self-evolution, the figure is forty-four point three eight percent. That gap of seven points is what self-evolution of the harness adds, on top of the same fixed executor.

All of the adaptation sits in the planner's harness, and none in the low-level controller. The harness carries the history a non-Markovian task needs, and self-evolution tunes that harness from rollouts before testing. The authors frame this as exploration, and suggest that agentic planning is a promising direction for manipulation that depends on history.