Grounded Action Model: 3D Grounding as a Foundation for Robotics

Grounded Action Model. 3D Grounding as a Foundation for Robotics. By Gehao Zhang and five others, from Northwestern University, the University of Washington, and two other institutions. Published on the twentieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

A manipulation policy has to know which objects matter and where exactly they are. Current robot foundation models are built on pretrained backbones that do not directly demand this metric grounding. For vision-language-action models, VLAs, the backbone comes from language. For world-action models, WAMs, it comes from video generation. Either way, grounding is left to be learned implicitly from robot demonstrations. The knowledge a policy most needs, which object and at what position in space, is the knowledge its pretraining never asked it to have.

Grounded Action Models, or GAMs, make 3D grounding the foundation instead of an afterthought. A GAM can be prompted with language, with points, or with boxes. Whatever the prompt, it is first transformed into one shared, object-centric representation of the objects selected.

That representation captures visual features focused on the target, together with the metric geometry of the objects. A multi-stream transformer mixes it with the history of the robot's state and predicts chunks of actions.

A GAM can run on its own. It can also serve as a low-level controller, taking direction from a high-level planner through any of its input modalities. That arrangement is what allows long-horizon and memory-dependent manipulation. The planner steers the grounded model through the same kinds of prompt it accepts from anyone else: words, points or boxes. Because every kind of prompt lands in the same object-centric representation, the planner is free to use whichever one suits it.

On RoboTwin two point zero, across fifty tasks, GAM averages a success rate of fifty-five point three percent, against fifty-two percent for Spatial Forcing. Under scene randomization it reaches forty-seven point six percent, against thirty point four percent for Abot-M0. Its action policy was trained only on clean-scene demonstrations. So the randomized result measures something the training data never showed it: randomized scenes, met by a policy that had learned only from clean ones.

On LIBERO-PRO, across sixteen perturbation settings, it averages a state-of-the-art sixty-one percent, against fifty-three percent for pi zero point five. The largest gains come when targets are relocated or newly designated.

On real hardware there are two robots. On a bimanual YAM under visual shift, GAM keeps seventeen successes out of twenty trials, where pi zero point five keeps four. Composed with a Molmo2 planner on a Franka, it achieves sixty-four point seven percent step completion in distribution and forty-nine point eight percent out of distribution, on long-horizon and memory-dependent tasks.

Across the results, the biggest margins appear exactly when the question is which object and where: scenes that are randomized, targets that move or are newly named, visuals that shift. In these comparisons, the model that grounds objects first holds up better under those changes than the baselines it is measured against.