Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?

Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?. By Zihao Zhang and seven others, from Southeast University, the University of Hong Kong, and one other institution. Published on the seventh of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Visual foundation models, or VFMs, are being folded into end-to-end autonomous driving for the strength of their representations. What has stayed unclear is when those representations actually improve driving. The answer here comes in three parts. VFM-guided representations do help, consistently. Which VFM you align to matters. And auxiliary perception supervision makes that choice matter much less. Put together, the answer to the title's question is a qualified yes. Better representations help, but how much depends on what they are aligned to and on how the planner is supervised.

The instrument is ViRA, a planner-agnostic visual representation alignment framework. It leaves the planner architecture and its inference cost unchanged, which is what lets one study span many planners. With it, the first finding is that VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, and the gains extend to zero-shot closed-loop evaluation.

The second finding is that the choice of VFM target matters for planning performance. Even planners that already have pre-trained VFM encoders can benefit further from alignment to a different VFM. The third concerns supervision. Auxiliary perception supervision reduces sensitivity to which target is chosen. Across five targets, the spread in EPDMS narrows from two point seven points to zero point five, and the authors suggest it may compensate for less effective targets.

These findings shaped a model. ViRA-Diffusion is a diffusion-based planner trained without auxiliary perception supervision, the setting in which target choice matters most. It reaches ninety-two point three EPDMS on NAVSIM v2 navtest, ahead of the recent methods in the authors' comparison by at least one point nine points. That margin is stated against the methods they compared, on that one benchmark split. The three broader findings likewise speak for the diverse planners, the five VFM targets and the evaluations the study ran, zero-shot closed-loop testing among them.

Better representations, then, are not a single dial. Alignment helps across planners, but how much it helps depends on which foundation model supplies the target, and that dependence shrinks once perception supervision is in the loop. The practical message the authors draw is to decide the target and the planner's supervision together, rather than one at a time, when bringing VFMs into end-to-end driving.