RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

RoboCasa-Shift. Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation. By Haocheng Wang and eight others, from Fudan University, the Chinese University of Hong Kong, and three other institutions. Published on the twentieth of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

The robustness of vision-language-action policies is usually summarised by a single number: aggregate task success. That number hides two things. It does not show how an individual execution changes when the distribution shifts, and in a long-horizon task it does not show where along the way the failures begin. Two policies, or two shifts, can produce the same drop in success for very different reasons.

RoboCasa-Shift is a diagnostic robustness benchmark built on RoboCasa365. It measures three things together. Outcome degradation, the familiar change in success rate. Paired outcome sensitivity, meaning how many individual episodes flip their result under a shift. And failure progression, meaning the stage of a long task at which execution breaks down.

Three design choices make those measurements possible. Shifts are controlled and single-factor. Rollouts come in matched clean and perturbed pairs, where each pair shares the same underlying settings and differs only in the target shift. And diagnostics are stage-aware and grounded in the simulator. The evaluation runs GR00T N1.5 on four long-horizon tasks that pass a predefined clean-performance gate, under four perturbation factors at three calibrated severity levels. Some conditions drive the policy to complete task failure. In total there are one thousand and forty formal rollouts.

Robustness responses vary substantially across factors and tasks, and raising severity does not always degrade performance monotonically. One camera condition leaves the change in success rate at exactly zero, even though eight of eighty matched episodes changed outcome. The aggregate saw nothing, and the pairs saw movement in both directions.

Stage-aware analysis adds a second layer. The same end-to-end degradation can correspond to quite different execution-depth profiles. Camera at level two and Gaussian blur at level one both reduce success rate by zero point one eight eight. Yet their first-stage failure rates are nineteen percent and thirty-six point four percent. Same score, different breakdowns.

The authors' conclusion is that robustness in these policies is a multi-dimensional property of execution, not a single end-to-end measure. That motivates evaluation protocols that track three things jointly: how much outcomes degrade, how sensitive paired outcomes are, and how failure progresses through a task. Each of the two headline examples shows a case where one of those three would have been invisible to the other two.