Agent as Policy for Robotic Manipulation
Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, Meng Jiang
University of Notre Dame · University of California San Diego · San Diego State University
Published September 11, 2026 · on the feed September 15, 2026 · ▲ 17 · arXiv 2609.12541
The abstract
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
The Brief
2:18A short podcast summary of the paper. Written and voiced by AI.
Transcript
Agent as Policy for Robotic Manipulation. By Mengzhao Jia and five others, from the University of Notre Dame, the University of California San Diego, and one other institution. Published on the eleventh of September. Its code repository is worked out: about one hundred and eighty source files, with a readme and a licence.
A robot policy decides what a robot does, moment to moment, while it carries out a task. The question here is whether a general-purpose agent can take on that role directly, driving a physical robot from the start of a task to its end, without any training specific to the task or to the environment. The authors demonstrate that it can. No policy is trained for the robot here. The agent itself is the policy.
Agent as Policy, or AGP, places both task planning and execution under the agent's control. The agent's reasoning and programming capabilities are brought into continuous interaction with the physical world, instead of stopping once a plan exists.
Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. That last step closes the loop: act, see what physically happened, and revise. Programs and motion commands are the agent's outputs, and camera evidence of what happened is its input on the next turn. Planning and execution stay in the same hands for the whole task, so an error seen in the world can be answered by a revised program.
AGP is studied across multiple real-world manipulation tasks, spanning precision manipulation, dynamic motions and deformable objects. The tasks include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. On three block construction configurations, AGP achieves success rates of one hundred percent, one hundred percent, and eighty percent. The spread of tasks is wide: exact placement, fast motion such as throwing, and cloth that changes shape as it is handled.
The authors see these findings as establishing a path for general-purpose agents to act as robotic policies. The agent's autonomy extends to physical manipulation through reasoning, programming and interaction at runtime, rather than through training for the particular job in front of it.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a robot that must handle tasks nobody trained it for.
You want a robot arm to assemble parts from a human video, flip dice and fold towels, with no training for any of these tasks or for this room.
The road ahead
- 1 · The problem ~7 min Robots that run a program fixed in advance, or that only pick among learned skills, cannot adapt both how they look and how they move.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min A general-purpose agent is itself the robot's policy for the whole task: it looks, writes code, moves, and revises after each outcome.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Given a task and a robot interface, the agent looks, writes programs, commands motions, and revises from physical outcomes until done.
- 6 · The experiment ~6 min You’ll guess: With no task-specific training, how often did the agent succeed at building block structures from goal images across three configurations?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.