RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents

RoboFollow. Unveiling the Instruction Following Mirage in Embodied Agents. By Chang Guo and eight others, from Shanghai Jiao Tong University and Anyverse Dynamics. Published on the twenty-second of September. Its code repository is worked out: about forty source files, with a readme and a licence.

Modern embodied agents report impressive success rates, yet their actual ability to follow instructions is far weaker than those numbers suggest. The authors trace the illusion to a structural property they call low scene entropy. When a visual scene admits only one valid task, the language instruction is redundant. A policy can score highly while barely using it, because the scene alone already says what to do. The success rate is real, but it measures something other than following instructions. It measures completing the one task the scene allows.

RoboFollow is a diagnostic benchmark built to measure genuine instruction following, which means forcing the policy to depend on language and then checking whether it does. Each of its design principles closes off a way that a policy could look competent without listening.

It rests on three principles. First, high scene entropy: every training scene supports several kinematically distinct task branches, so vision alone is not enough and the instruction must be used. Second, a hierarchical diagnostic protocol with four levels, L0 to L3, that progressively perturb visual layout and semantics. It probes whether equivalent instructions produce consistent behaviour, and whether distinct ones produce behaviour that can be told apart, across spatial relations, attributes, trajectory constraints, and logic.

Third, confound-controlled diagnosis. Interaction objects are simplified, actions are restricted to the trained repertoire, and results are reported as separate Intent and Execution scores at each stage. That isolates understanding the instruction from carrying it out physically.

Nine VLA and WAM policies were evaluated, each fine-tuned under the same setup. Strong performance at L0, where it was reached at all, does not reliably carry over to L1 through L3 under the authors' fine-tuning setup. Representative mitigations were tried: stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance. All of them fail to close the gap. That is the sharper result. Under this setup, the shortfall is not fixed by a stronger language backbone, by extra question-answering data, or by guidance at decoding time.

The benchmark's point is that a high success rate can be earned without listening. Once a scene stops giving the answer away, the authors find instruction following to be a critical and overlooked bottleneck. Code and dataset are public.