VITAL: From Visual Motion to Precise Action

VITAL. From Visual Motion to Precise Action. By Ao Liang and nine others, from Horizon Robotics, the Shanghai Innovation Institute, and three other institutions. Published on the fourteenth of October. Its code repository is worked out: about fifty source files, with a readme and a licence.

World-action models try to turn visual anticipation into driving decisions. Yet the action experts inside them are typically trained on trajectory representations that are defined independently of visual dynamics. What the model sees moving and what it is asked to output are described in unrelated terms. That motivates a different target: derive the actions directly from visual motion.

Before building that, the authors test a simpler setting. Trajectories are reconstructed from recorded future images, so there is no need to predict the future at all. Even then, reconstruction captures the dominant maneuver but leaves errors at the scale of centimetres. Visual motion gets the gist of the trajectory right, and misses the fine detail.

VITAL, short for Vision-Induced Tokens for Action Learning, is an action tokenizer built around that observation. It couples two pieces. One is a motion scaffold derived from vision. The other is a compact residual that encodes the trajectory error the scaffold leaves behind. Visual motion becomes the principal thing to predict, and the small metric corrections get a separate, normalized learning scale of their own.

An action expert trained with flow matching predicts both components together. Its inputs are the current image, the ego state, and the navigation command, and it does not generate future images at inference. The residual coordinates have a second use. They support bounded reinforcement post-training, which refines execution while the predicted visual scaffold is held fixed.

Results are reported on NAVSIM. With only one action denoising step, VITAL scores ninety point eight PDMS on the version one navtest split. That is within zero point one three points of its own ten-step score, at three point two nine times the end-to-end inference speed. Without retraining, it scores ninety-one point one EPDMS on the version two navtest, and thirty-eight point six on navhard.

A division of labour runs through VITAL, and the reconstruction experiment is what made it visible. Vision is good at the broad shape of a manoeuvre and short of the last few centimetres. Giving each its own representation, and its own scale, lets the model predict mostly what it can see, and correct the rest separately.