DriveZero: End-to-End Driving Beyond Human Demonstrations

DriveZero. End-to-End Driving Beyond Human Demonstrations. By Hao He and nineteen others, from Xiaomi EV. Published on the fifth of September. Its code repository is worked out: about four hundred and fifty source files, with a readme.

Most end-to-end autonomous-driving systems learn by imitating logs of human driving. That ties what they learn to the recorded trajectories: to how good those trajectories were, and to how much behaviour they cover. A system trained this way can do no better than the demonstrations allow. Whatever the drivers in the logs never did, the model never sees. DriveZero is an end-to- end system meant to learn driving behaviour beyond those human demonstrations.

It decomposes driving into a perception model and an action model, pretrains each in the regime best suited to it, and then combines them into a single end-to-end planner. The two call for different recipes. Perception has to understand the world, and benefits from massive and diverse visual data. Action has to interact with the world, and that requires closed-loop feedback. Pretraining them in separate regimes lets each get the kind of learning signal it needs, instead of forcing both through one recipe.

On the action side, DriveRL is a mixed-agent, closed-loop reinforcement- learning framework. It converts real driving logs into interactive worlds, and in those worlds a privileged teacher policy is trained with PPO through closed-loop rollouts. The teacher is then frozen, and that frozen policy is what the final planner learns from. On the perception side, DriveVFM consolidates several frozen vision foundation models, including DINOv3, SigLIP2, SAM and Depth Anything V2, into a single backbone. It learns from raw images alone, with no task-specific annotations.

DriveZero then unifies the two. It is a camera-only planner that distills the frozen DriveRL teacher through the trajectories that teacher rolls out. Because the teacher is goal-conditioned, it can also be queried under augmented driving intents. That yields diverse supervision that stays consistent with each goal, supervision that logged data cannot provide.

On nuPlan, DriveRL with value-guided test-time action search reaches a mean score of ninety-three point five seven across three community splits: Val14, Test14-hard and Test14-random. That holds in both non-reactive and reactive modes, and it exceeds the Log-Replay expert on all three splits. The full DriveZero system achieves state-of-the-art performance on NAVSIMv1, NAVSIMv2, and the closed-loop HUGSIM benchmark, all without any human trajectory supervision. Two levels of result, then. The teacher itself outscores replaying the logged expert, and the camera-only student that learns from it reaches the top of three benchmarks, one of them closed-loop.

The design separates what can be learned from looking and what has to be learned from acting. Perception gets its data from foundation models and raw images, action gets its experience from logs turned into interactive worlds, and the final planner inherits from a teacher that was never limited to copying people. Exceeding the Log-Replay expert is the clearest sign of what leaving human demonstrations behind allows.

DriveZero. End-to-End Driving Beyond Human Demonstrations. By Hao He and nineteen others, from Xiaomi EV. Published on the fifth of September. This is the long version: the whole paper rather than the abstract.

Most end-to-end driving systems learn by copying human driving logs. That makes the recorded human trajectory the only source of supervision, and the Xiaomi EV team names three ways that bites. Each logged scene contains only one realized future, even when several actions would be valid. Safety-critical deviations and recovery maneuvers are rare. And the states a learned policy drives itself into are simply absent from the offline data, so errors compound once the policy leaves the demonstrations. Reinforcement learning fixes all three in principle, because a policy that acts in closed loop sees the consequences of its own mistakes. The catch is cost: learning to drive from cameras by trial and error couples sample-hungry exploration with expensive visual simulation.

The way out is to split driving in two and train each half where it learns best. Perception has to understand the world, and benefits from huge and varied image collections. Action has to interact with the world, and needs high-throughput closed-loop feedback, which is cheap only over compact structured state: positions of cars, lane geometry, traffic lights. So the action model learns from structured state in a fast simulator, the perception model learns from images with no driving labels at all, and a third stage reunites them. The bridge is the driving log itself. Every logged frame pairs the camera images with the structured state of the same moment, so a trajectory the teacher produces from the state can be used to supervise a student that only sees the images.

The action model is DriveRL. It starts from random weights, with no imitation pretraining: logs only seed the scenes and the navigation goals. Each background actor gets its own behavior provider through one common interface, so log replay, a rule-based car-following model called IDM, sudden front-vehicle braking, and learned self-play agents can share a scene. The policy sees nearby traffic, the local map and two goal points, and outputs bounded distributions over jerk and steering rate. The reward has a telling shape. Hard events like collisions end the rollout. Reaching the goal earns a one-time bonus. Six soft driving-quality scores are multiplied together rather than added, so excellence on one cannot hide a serious failure on another. Training runs two thousand and forty-eight worlds per GPU on ninety-six GPUs, for about twenty-one hours.

At inference DriveRL can also search. It keeps its usual action as candidate zero, samples alternatives from the same policy, rolls each forward five steps, and scores them with the training reward plus the critic's estimate of what follows. It switches only if an alternative beats the default by a fixed margin, because both the short rollout and the critic are approximate.

The perception model is DriveVFM, a single vision transformer trained to match the frozen features of four foundation models, each standing in for a family of perception tasks. DINOv3 supervises both its image summary and its patches, SigLIP2 only the summary, SAM only the patches, and Depth Anything through its features, which worked better than distilling its depth maps. Since no labels are needed, web imagery and driving scenes mix freely. DriveZero then encodes four camera views with DriveVFM, proposes sixty-four trajectories, and is trained winner-takes-all: only the proposal closest to the teacher's trajectory is pulled toward it. A separate scoring head learns to rate each proposal and picks the winner at inference. Finally, because the teacher is goal-conditioned, the same scene can be replayed under different driving intents, turning one logged moment into several valid targets.

On nuPlan, DriveRL averages ninety-three point zero one across six closed-loop settings and beats the Log-Replay expert in every one: the policy outdrives the logs that seeded its worlds. Search lifts the mean by about half a point, to ninety-three point five seven, with one setting essentially flat. On NAVSIM, the camera-only DriveZero ties the human driver at ninety-four point eight, and with extra simulated scenes reaches ninety-five point three. On the true closed-loop HUGSIM benchmark, never seen in training, it scores forty-six point six against a previous best of thirty-eight point five.

The comparison that most directly tests the thesis sits in an ablation. Trained on teacher trajectories alone, the student scores ninety-three point six one, slightly below the same student trained on human trajectories at ninety-three point nine two. Only with goal augmentation does it rise to ninety-four point four one and pass the human baseline.

The report has no limitations section, so the limits here come from its own appendix. Self-play, the configuration used for real-world deployment, gave comparable scores but no consistent advantage on nuPlan; the authors attribute this to the benchmark driving its background traffic with IDM, which does not reward more natural interaction. A red-light reward cut the policy's violation rate from three point eight eight percent to two point zero three percent, yet lowered all six benchmark scores, because traffic-light violations are not part of the score. Several key gains are fractions of a point on what the authors call a highly saturated benchmark, reported without repeated runs or error bars. The real-world claim is a small fleet and a demonstration video, with no reported numbers, and it is the structured-state DriveRL on the road, not the camera-only DriveZero.

The idea to carry away is that the most valuable property of a reinforcement learning teacher here is not that it drives better than a person on any single trajectory. It is that it can be queried. A log records one future per scene; a goal-conditioned policy can supply as many futures as there are sensible intents, each consistent with the scene. Pair that with perception learned from images alone, and an end-to-end planner can be trained without a single human trajectory as its target.