PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

PerturBot. Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training. By Mingyu Liu and six others, from Zhejiang University, the Shanghai Innovation Institute, and two other institutions. Published on the third of October. Its code repository is worked out: about ninety source files, with a readme and a licence.

A vision-language-action policy can complete complex tasks while ignoring the evidence that ought to decide its actions. The examples are concrete, and they cover all three modalities. In vision, an object held near the wrist camera can displace the target the instruction named. In language, a familiar noun can trigger the operation it was paired with in training, even after the verb in the instruction has changed. In action, a gripper that closed on nothing may lift anyway, as though it were holding something.

The authors call these modality shortcuts. Regularities in successful demonstrations make a visual, lexical, or motor cue enough to predict the expert's action, without the task evidence the decision actually depends on. In the training data, the cue and the right action always went together, so the policy never had to tell them apart. That is why more demonstrations of the same kind can raise task success while leaving the shortcuts exactly where they were.

Perturbot aims to make task-relevant evidence easier to use, and to make the shortcuts insufficient on their own. It complements scaling rather than replacing it: instead of adding more of the same data, it changes what is scaled. Inference is left unchanged, so the whole intervention lives in training.

It works on the same three channels where the shortcuts appeared. On vision, it applies wrist-view perturbations that preserve the task. On language, it enriches instructions with captions that are relevant to the decision. On action, it adds random and failed trajectory segments, each relabeled with the behavior it actually contains, so that a closing gripper is no longer always followed by a successful lift.

Alongside the training recipe comes a measurement. GroundingFscore is an offline score that diagnoses how severely a policy relies on modality shortcuts.

The abstract frames the evidence through that pairing rather than through reported figures. Task success rate shows whether a policy improves. GroundingFscore shows whether it is scaling healthily, leaning on task evidence rather than on shortcuts. A policy can climb on the first while standing still on the second, which is exactly the situation the authors describe when more of the same demonstrations are added.

Perturbot and GroundingFscore are offered together as a training-and-evaluation framework. One half separates the policy's decisions from shortcut priors while keeping it responsive to the evidence that matters. The other half makes that separation visible, so that success alone is no longer the only reading of whether a policy has learned the task. Each half needs the other: a fix nobody can measure, or a diagnosis with nothing to apply it to.