Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Not All Features Are Created Equal. A Mechanistic Study of Vision-Language-Action Models. By Bryce Grant and two others, from Case Western Reserve University. Published on the twentieth of October. Its code repository is worked out: about one hundred and sixty source files, with a readme and a licence.

A fine-tuned vision-language-action policy can place the alphabet soup in the basket on command, just as it did during fine-tuning. Shift the basket five centimeters to the left, and it drops the soup off the table. The authors built an interpretability pipeline to find out what such a policy is actually computing, and their central claim is uncomfortable. The program the policy runs is bound to absolute workspace coordinates, and it ignores the visible scene. That, they argue, is a mechanistic account of the perturbation brittleness that concurrent benchmarks report.

The core experiment is an injection. Take task A's activations, put them into the action expert while the policy is looking at task B's scene, and pi zero point five executes task A's motor trajectory in ninety-nine point six percent of episodes, a figure drawn from a large sample of runs. What it sees does not override what was injected.

The binding survives on a physical UR10e arm. Directed replay redirects the commanded task in twenty-six out of twenty-six rollouts in one direction, and eighteen out of twenty-six in the other, while no-injection and random-direction controls stay at the floor. Then the telling test: move the source object ten to fifteen centimeters. In fifty out of fifty trials the arm grasps and releases at the recorded coordinates anyway. It picks up the wrong block when one happens to sit there, and closes on empty air when none does.

The same interventions were then applied to six VLAs on LIBERO, MetaWorld, SimplerEnv and ALOHA, over hundreds of thousands of rollouts. Three further findings come out. Every architecture encodes the instruction, yet four of the five weight it below vision whenever the scene alone identifies the goal. Under the standard replacement write-back, which SAE pooling preserves behavior depends on the architecture: per-token for pi zero point five, while SmolVLA is indifferent. And goal selection and motor generation separate into the VLM pathway and the expert pathway, wherever the two can be told apart.

The limits need stating with care. The sharpest results, the injection rate and the physical replay, are reported for pi zero point five on one physical arm. The cross-model findings come from six VLAs on four named suites, and two of them are qualified in the abstract itself: the instruction is weighted below vision in four of five architectures, not all, and only when the scene alone already identifies the goal. The pathway split is claimed only where the two pathways are distinguishable. What the work offers is an account of one kind of brittleness, not a general theory of how these policies fail.

Alongside the paper comes Action Atlas, an interactive platform covering all of the more than five hundred and twenty trained SAEs, seventy-nine identified concepts, and the scored UR10e rollouts. The image to keep is the arm closing on empty air: a policy faithfully replaying where things used to be, with its eyes open and its attention elsewhere.