PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

PointWAM. 3D World Action Modeling for Dexterous Robotic Manipulation. By Chunghyun Park and seven others, from POSTECH, KAIST, and two other institutions. Published on the second of October. No code repository is linked to it.

A world action model learns two things jointly: how the world will evolve, and what the robot should do, with the first guiding the second. Existing approaches usually picture the world as RGB frames, or as latent versions of them, and express actions as end-effector poses or joint angles. For dexterous manipulation that combination often falls short. What matters most there is the 3D spatial structure of the scene and the geometry of contact, and neither flat frames nor joint angles capture those well.

The Point World Action Model, PointWAM, moves the whole problem into three dimensions. It splits the world into two parts: the scene, meaning the environment, and the hands, meaning the actor. Both are forecast jointly as 3D point trajectories, inside one coordinate frame shared across space and time. Scene and hands are kept apart as representations, yet predicted together in the same space.

That explicit, disentangled representation is what opens the door to pre-training on large-scale videos of human demonstrations, without having to select task-specific objects or keypoints. At run time, PointWAM takes a colored point cloud and a language instruction, and predicts how the scene and the hands will co-evolve in 3D over time. The forecast hand motion is then retargeted into robot actions. The robot's moves are read off a forecast of hands, which is the same kind of thing the human videos show.

Each ingredient is measured. Pre-training on human videos improves average success on DexJoCo by fifty-six point nine percentage points. Supervising the scene trajectories as well adds a further ten point nine points over forecasting the hands alone. With both in place, PointWAM surpasses the previous state of the art on ten DexJoCo tasks by eleven point seven points. On a real robot, it outperforms strong VLAs.

Both measured gains trace back to one representational choice. Splitting actor from environment, and forecasting each as points in a shared 3D frame, is what the authors credit with making large-scale human demonstration video usable for pre-training, and the scene half earns its own margin on top of the hands. One model learns from people moving their hands, and a robot acts on what it forecasts.