SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

SimVTA. Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models. By Changhui Jing and ten others, from the Shanghai Innovation Institute, the University of Hong Kong, and three other institutions. Published on the ninth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Driving world action models, or WAMs, learn from trajectories paired with the future observations that followed them. Post-training with reinforcement learning lets the policy explore alternative trajectories and improve its planning against driving rewards. The trouble is the data. A driving log contains future video only for the trajectory that was actually driven. Every alternative the policy explores has no matching visual supervision at all. The world model half of the system can learn from what happened, but not from what might have happened, which is exactly what exploration produces.

SimVTA is a simulation-grounded post-training framework that supplies the missing video. It combines trajectory optimization with video supervision, and the video comes from a simulator rather than from the model itself. That gives the alternatives trajectory-matched supervision that is independent of the model's own predictions.

For each observation, candidate trajectories are sampled. A real-world simulator, World Engine being the example given, renders future videos for the high-reward candidates, inside real-world scenes reconstructed with 3D Gaussian Splatting. Rendering is limited, so under that budget the framework prioritizes video supervision for the futures with higher error among the rendered candidates.

Training then pulls on two losses. Group Relative Policy Optimization, or GRPO, optimizes trajectory planning. A flow-matching loss on the matched videos updates the representations the two tasks share. At inference time none of this machinery is needed: the model predicts trajectories directly and makes no simulator calls.

On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from eighty-six point two to ninety-one point three, a gain of five point one. It reaches eighty-eight point two EPDMS on NAVSIM v2. In closed-loop evaluation on HUGSIM, it records an average HD-Score of thirty-eight point eight.

Ablations separate the two ingredients. Updating shared representations with video supervision improves planning on its own. Prioritizing higher-error futures among the high-reward trajectories improves performance further. Both design choices, in other words, are shown to carry weight: the shared representation as the channel through which video reaches planning, and the error-based priority as the way a limited rendering budget is spent.

Reinforcement learning asks the policy to imagine roads not taken, while the logs can only show the road that was. SimVTA closes that gap by rendering the promising alternatives, spending its limited rendering where the model is most wrong, and letting those videos shape the representation the planner relies on. The simulator is a training-time teacher only, absent once the model is deployed.