In-Context Robot Learning with VLM Agents

In-Context Robot Learning with VLM Agents. By Dongzhou Cheng and fourteen others, from Morphi Robot, the Shanghai Innovation Institute, and eight other institutions. Published on the sixteenth of September. Its code repository is worked out: about eighty-five source files, with a readme and a licence.

Robots that adapt to unfamiliar environments as readily as people do remain a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will meet, so the ability to learn from context at deployment time is essential for generalizing. That kind of in-context learning, ICL, is still largely out of reach for existing robot policies. A policy learns what it was trained on, and adapting it usually means training it again.

Commercial vision-language models, VLMs, such as GPT-6 Astra, have broad agentic capabilities, and that raises a question. Can such a model learn from demonstrations, examples, and feedback from interaction, and then turn what it has learned into robot behaviour that is executable and verifiable, starting from a new initial state? And can it do so without gradient updates, and without persistent changes to parameters specific to the task?

GPT-Policy is a general-agent framework built to test that. Learning happens entirely in context: what the robot is shown and told becomes the context for a VLM, and the VLM's proposals become the robot's actions, with a controller standing between the model and the hardware. The question it is built around is exactly the one above. Can the model be shown, told, and corrected, and then act on that, with nothing about it retrained?

The framework has three parts. A context compiler keeps the visual transitions that matter for the task, so the context holds what changed and not just what was seen. A VLM proposes actions in terms of robot tools. And a constrained controller verifies each proposed action, executes it, and reports the outcome back. The division of labour is clear. The model reasons about what to do. The controller decides whether a proposed action is allowed and carries it out. The report of each outcome is feedback the model can learn from in the next step, all within the context.

The authors evaluate reliability and limitations in three ways: task success and efficiency metrics, matched comparisons across models, and controlled ablations of the context. In trials on a real robot, human video demonstrations improve task completion even without any robot action labels. Adding aligned action references gives further gains on tasks sensitive to contact. The first result says the robot can learn from watching people, with no robot-specific labels at all. The second says that when touch and contact matter, showing it actions that line up with its own adds more.

GPT-Policy is offered as a step toward robots that adapt by learning in context. It offers an empirical foundation for turning the general-purpose abilities of VLMs into physical behaviour, and it is equally meant to clarify what still has to be overcome before such systems can be deployed reliably. The framework is presented as a way to find where in-context robot learning works and where it does not yet, not as a finished answer.