HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

HexaShell. A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents. By Hongcheng Gao and fifteen others, from HexaFuture. Published on the sixteenth of October. Its code repository is worked out: about three hundred source files, with a readme and a licence.

Agents built on vision language models now compete with learned policies on robot manipulation leaderboards. That raises a plain question, and the paper asks it directly. A coding agent is used to controlling a computer. How far can it go when what it controls is a robot instead, and where does it stop? The paper is framed as an exploration of that limit, so finding the stopping point matters as much as the score.

HexaShell answers by keeping the harness thin. Its only job is to turn the agent's commands into robot motion. Everything else is left to the agent, which works the way a coding agent works on software. After each failed attempt, it reads the log, finds the cause, and writes or fixes small programs the authors call tools. What it learns is kept as code. The analogy the authors draw is to a coding agent fixing a failing test: the failure is read, the cause is found, and the fix stays in the codebase for next time.

The agent drives a robot with two arms through a single command line. It sees three RGB cameras and knows their calibration. The tools it writes may use only what the robot observes, and they locate objects by triangulation, both relative to one another and relative to the gripper. The geometry is simple enough to say in two sentences. If two cameras see the same point, the point lies where the two camera rays meet. If a point lies on a known plane, like the table or the top of a block, it lies where its ray meets that plane. And when the head camera is not enough on its own, the robot moves a wrist camera to get a second view.

The protocol is set up to test whether the tools generalize rather than memorize. Tools are built on only ten layouts of one seed per task. They are then frozen, so no further fixes are allowed, and tested on ten new layouts of that same seed. The agent itself is Codex, running GPT-6-Astra at medium reasoning effort.

Across all forty-two RoboDojo tasks, HexaShell reaches a success rate of thirty-nine point seven percent and a progress score of forty-eight, averaged the same way the leaderboard averages them. That makes it a new state of the art on RoboDojo, achieved with a harness that does nothing but relay commands.

The breakdown by task type is where the paper's real finding lives, because the average hides a wide spread. Pushing reaches ninety-two point five percent. Memory tasks reach sixty-five percent. Tight insertion reaches only eight point three percent. The failures have a pattern, too. When the agent fails, it usually sees the failure. What it cannot do is make the motion work. Perception and diagnosis hold up, and the physical execution is what gives way.

HexaShell's own summary is the line to carry away: the limit is contact, not reasoning. A coding agent with cameras and a command line can diagnose, locate and plan well enough to lead the leaderboard, and it still cannot reliably complete a tight insertion. That puts the open problem squarely in contact-rich motion. The tools evolved so far are open source, so others can start from them.