RobotUse: Allocating Computation, Context, and Decisions

RobotUse. Allocating Computation, Context, and Decisions. By Junhoo Lee and six others, from KAIST and Seoul National University. Published on the fourth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

A robot agent has to connect what it meant to do with what actually happened, and it has to keep enough context to revise its choices across repeated attempts. Both halves matter. Without the first, it cannot tell why an attempt failed. Without the second, it cannot do better on the next one.

Existing interfaces tend to fail in one of two directions. Some leave the important choices buried inside predefined tools, out of the agent's reach. Others hand the agent everything, so it must manage detailed execution code along with a history that keeps growing.

RobotUse is a robot agent harness that organizes computation, context, and decisions around one activity: specifying physical actions and revising them.

Each of the three is placed deliberately. Computation is split: the agent selects targets and poses visually, while the backend takes care of geometry, motion planning, and control. Context is contained: subagents keep the detailed interactions inside each subgoal and hand back only what later decisions need. Decisions improve over time through continual harnessing, where the agent learns from execution by updating a persistent playbook.

On RoboLab, RobotUse reaches forty-five percent task success, six point seven percentage points above CaP-X. It does so while keeping decision contexts compact and depending less on predefined action abstractions. The authors also show that it learns from real-world execution despite imperfect feedback, and that it carries what it learned over to later tasks.

Where each piece of work lives is the design. Low-level control goes to the backend, fine detail stays with subagents, and lessons go into a playbook that persists. The agent is left with the part only it can do: choosing what to do next, and revising that choice.