World Action Learning via Interaction-Centric Spectral Latent Guidance

World Action Learning via Interaction-Centric Spectral Latent Guidance. By Zhiming Liu and nine others, from the Hong Kong University of Science and Technology. Published on the second of October. No code repository is linked to it.

General-purpose robot policies need large amounts of real-world interaction data, and robot demonstrations are expensive and hard to collect at scale. Egocentric video, filmed from a person's own point of view, is abundant by comparison, and it is full of human interaction carrying the semantics that robotic manipulation tasks need. Transferring it directly runs into two obstacles. Latent actions inferred by reconstructing frames can be dominated by nuisance variation, ego-camera motion above all. And humans and robots often move with different temporal dynamics, so even the right action can unfold on a different clock.

WING, short for World Action Learning via Interaction-Centric Spectral Latent Guidance, takes on each obstacle in turn. Against the first, it separates the motion induced by the observer from the interaction between hand and object, and distills only the interaction-centric part into latent actions. Against the second, it relies on an observation: the task semantics that carry across embodiments are concentrated in temporal structure that varies slowly.

The first step addresses what the latent actions contain. Frame reconstruction alone can let the movement of the camera itself pass for action, so the observer's own motion is pulled out before anything is distilled. What remains describes the hands and the objects they act on, which is the part of a human video a robot could hope to reuse.

The second step addresses timing, and it is put to work in the spectral domain. WING identifies the low-frequency components that egocentric latent actions and robot behaviours share, and uses those shared components to guide action generation. The fast details, where a human hand and a robot arm differ in timing, are not what the guidance is built from. The slow structure, where the task itself lives, is.

WING reaches an average success rate of ninety-nine point two percent on LIBERO. On RoboTwin 2.0 it reaches ninety-three point eight percent, and on RoboCasa-GR1 fifty-seven point seven percent. It also performs strongly across four real-world manipulation tasks, under diverse generalization settings.

The authors conclude that interaction-centric spectral guidance is an effective and scalable way to move physical interaction knowledge from human egocentric video into robot control. The two steps answer two different mismatches. Strip the camera's motion so that what remains is the interaction, then keep the slow part of that interaction, because the slow part is what a person and a robot have in common.