Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Breaking the Vision-Action Shortcut. Latent Interface Training for Generalizable Robotics Foundation Models. By Jianman Lin and three others, from the South China University of Technology, the National University of Singapore, and one other institution. Published on the eleventh of September. Its code repository is public and real: about twenty-five source files.

Robot foundation models perform strongly in distribution but often degrade under visual distribution shifts. The explanation offered is a shortcut. When a model learns to generate actions from pretrained visual representations, it may exploit task-irrelevant visual cues that happen to correlate with the demonstrated actions within the training distribution. When those correlations change, generalization breaks. Fixing this means constraining how visual information is used for action generation, without losing the spatial information the task actually needs.

Latent Interface Training, or LIT, is a two-stage strategy that is agnostic to the underlying framework. It first establishes an action prior with no images, conditioned on a spatial goal. Only after that does it let vision in, through a single latent interface that is supervised with pose. The order is the point. If the action expert already knows how to act once it knows where it is going, then the job left for vision is narrow: supply that goal. The interface is supervised toward exactly that job, by reconstructing the goal pose.

In Stage 1, the action expert learns to generate action chunks conditioned on language, robot state, and the terminal end-effector pose of each demonstrated chunk, expressed in SE three. It learns goal-directed action generation independently of visual cues. With no images in the input at all, there is nothing in this stage for a visual shortcut to attach to.

In Stage 2, a latent interface aggregates visual and semantic representations, and becomes the pretrained action expert's only visual conditioning pathway. The interface is supervised to reconstruct the same terminal pose that conditioned the first stage. That encourages it to retain the goal-relevant spatial information action generation needs, and gives other cues little reason to pass through.

LIT was applied to four vision-language-action and world-action architectures: Pi0.5, MolmoAct2, FAST-WAM and ImageWAM. Across them it improves overall success on LIBERO-Plus by between three point eight seven and ten point seven percentage points, while preserving or improving average success on LIBERO. In real-world evaluations, success aggregated across three tasks rises by between thirteen point three and sixteen point seven percentage points, under unseen camera configurations, lighting variations and distractors. Those are exactly the kinds of visual change that a shortcut would be expected to trip over.

The strategy narrows the path vision can take. When images can reach the action expert only through a representation trained to predict where the end-effector should finish, a background cue has far less opportunity to become a shortcut.