AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation

AffordanceWAM. Affordance-Aware Joint World-Action Modeling for Robot Manipulation. By Jiadi You and thirteen others, from the University of Hong Kong, Peking University, and four other institutions. Published on the eighteenth of October. Its code repository is worked out: about one hundred source files, with a readme and a licence.

A robot that manipulates well in general has to do three things. Predict how a scene will evolve, identify where interactions are feasible, and decide how to act. The data for learning this comes in two kinds, and each falls short. Robot videos labeled with actions supervise control directly, but they are costly and limited in diversity. Egocentric human videos capture a wide variety of interactions, but they have no robot actions, and the human body and appearance differ from the robot's.

AffordanceWAM is a generative World Action Model that puts affordance at the centre. It represents where and how objects can be interacted with, over space and time, through two signals: a Scalar Affordance and an Affordance Heatmap, both living inside the generated future World. This does two jobs. It grounds visual prediction in the objects and regions that matter for the task, which is what action generation needs. And it gives human and robot videos a shared interaction target, something both kinds of data can speak to.

The model is built on a pretrained video diffusion Transformer. It has separately parameterized World and Action Experts, coupled through Masked Joint Self-Attention. Together they jointly predict four things: future RGB observations, Scalar Affordance fields, Affordance Heatmaps, and continuous robot actions, all under one unified flow-matching objective. The two data sources plug in differently. Human videos supervise all three future-World streams. Robot trajectories supervise those too, and additionally provide action supervision. That split is what lets the model transfer from human video without any human action labels and without retargeting.

Experiments cover RoboCasa, the CALVIN benchmark in its ABC to D setting, and real-world manipulation. Across them, AffordanceWAM shows consistent gains over baselines that use RGB only and baselines that use robot data only. One result speaks directly to scaling. With robot supervision held fixed, RoboCasa performance improves monotonically as the amount of affordance-annotated human video grows.

The authors read this as support for affordance as an interface, one serving two purposes at once: vision-language-action learning, and transfer from humans to robots. Human video cannot say what motor command to send, but it can say where an interaction happens and how the scene will change, and in these experiments that common ground was enough for more of it to keep helping.