Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

Foundation4D. Towards a unified perception and reconstruction model for autonomous vehicles. By NVIDIA and twenty-six others, from NVIDIA. Published on the twenty-first of October. No code repository is linked to it, though it does publish its weights.

A car with several cameras needs to know a great deal from the same video at once: the geometry of the scene, where the objects are, which space is occupied and by what, and how to reconstruct the scene for later use. Each of those is a separate problem in its own right, and each draws on the same stream of images from the same cameras over the same stretch of time. The aim here is to answer all of them with a single model.

Foundation4D is a unified feed-forward model that handles geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. One shared backbone serves every task, and it uses factorized spatiotemporal attention to combine observations efficiently across cameras and across time.

For video inference, a pipeline applies pose graph optimization to refine the camera trajectories, and assembles object tracks across input windows. The authors also built a data annotation pipeline, and with it release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for one thousand two hundred and fifty-three driving clips.

With fine-tuning for each benchmark, Foundation4D has the lowest depth AbsRel on Waymo and on nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy, among the methods compared. A study of task combinations shows the tasks helping each other. Training detection together with geometry and occupancy lifts top-class mAP by one point two, compared with detection alone. Training geometry together with detection cuts camera-to-rig translation error by thirteen point four percent.

The predicted geometry and trajectories also serve as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation. So the outputs of the perception side feed the reconstruction side, without lidar. The case being made is that unification pays twice: one backbone instead of several, and tasks that sharpen each other when trained together, with detection gaining from geometry and geometry gaining from detection.

Foundation4D. Towards a unified perception and reconstruction model for autonomous vehicles. By NVIDIA and twenty-six others, from NVIDIA. Published on the twenty-first of October. This is the long version: the whole paper rather than the abstract.

A self-driving team wants two things from the same camera footage. It wants to understand the road: where the surfaces are, where the cars and people are, and which space is occupied. And it wants to rebuild the recorded drive as a scene it can render from new viewpoints, for simulation. Both jobs rest on the same facts: geometry, objects, and how they move. Yet they are usually handled by separate networks, each reprocessing the same frames. Foundation4D, from NVIDIA, puts all of it into one feed-forward model. It reads six cameras over time and, in a single pass over a window, predicts camera poses, metric depth, three-dimensional boxes, semantic occupancy, and renderable Gaussians.

Two pieces of background carry the design. The first is VGGT, a geometry model that recovers cameras and dense depth from a set of images by alternating attention within each image with global attention across all of them. That global attention grows quadratically with the total number of image tokens, so six cameras over many timestamps gets expensive quickly. The second is the physical structure of a car. Its cameras are bolted to one rig. Every camera's pose is therefore a fixed offset, the rig calibration, composed with one moving trajectory, the ego motion. A model that knows this has far fewer things to estimate than one that treats every image as an independent camera. One wrinkle remains: the cameras do not all fire at the same instant, so each image keeps its own capture time.

Each image passes through a shared DINOv2 encoder. Its patch grid is shrunk to a quarter of the tokens before aggregation, while the full-resolution features are kept aside for detection and occupancy. The aggregator then factorizes the global attention. Every layer attends within an image, then either across cameras at one timestamp or across time for one camera, and the two kinds alternate. Each image also carries a rig token, an ego token, and four register tokens. A rig head reads the rig tokens to recover each camera's calibration. An ego head reads the reference camera's tokens across time to recover the trajectory, which is interpolated to each image's own capture time. Depth comes from dense decoders, and the final point cloud is built by casting that depth out along calibrated camera rays. Depths and translations are divided by a fixed reference length of forty metres before supervision, which pins every window to the same metric scale.

The other branches hang off the same features. Detection uses nine hundred queries anchored at learned points in space, which predict boxes with a planar velocity. Occupancy keeps its features in a bird's-eye-view grid, adding height only at the output, to save memory. The Gaussian branch predicts one Gaussian per pixel, placed using predicted depth and the predicted trajectory, and attaches actor Gaussians to tracked boxes.

Long drives exceed one window, so overlapping windows must be reconciled. Simple stitching aligns each new window to the previous one, so errors pile up. Pose graph optimization instead makes every reference-camera sample a node, uses each window's relative poses as constraints, and adds a prior against abrupt velocity changes. Because training fixed the metric scale, no per-window scale has to be estimated. Detections are then moved into this common frame and tracked offline, with a backward smoother that lets later observations correct earlier ones.

After fine-tuning for each benchmark, the headline results hold: the lowest depth error on Waymo and nuScenes, sixty point nine mAP on nuScenes detection, and eighteen point one mIoU on OpenOccupancy against sixteen point two for the strongest baseline. The ablations say more about why. On fifty internal clips, factorized attention on its own cut runtime by eight point five percent but made trajectory and depth slightly worse. The largest single gain came from metric-scale training, which roughly halved trajectory error, from about one hundred and forty-two centimetres to about seventy-one.

Pose graph optimization then nearly halved it again compared with masked stitching. Detection gained about ten points of mAP simply from feeding thirty-two timestamps instead of two, with no retraining. And the Gaussian branch, using only its own predicted poses and tracks, reached twenty-three point nine four decibels of PSNR overall, against twenty-two point six six for Instant NuRec, which was given ground-truth poses.

The paper has a limitations section, and it starts with exactly that result: more tasks do not always help, and better task weighting may fix it, but testing new recipes is costly. Perception training took about eighteen days, and Gaussian training another nine. The authors have not tried training the Gaussian branch jointly from the start, nor studied how geometry and rendering supervision affect each other. Depth degrades at long range and around object edges, rig supervision can mix calibration with motion between exposures, and occupancy misses rare classes and flickers between frames.

The evidence has its own edges. The Gaussian win is in static regions. When extrapolating to future frames, dynamic regions score lower than Instant NuRec, fifteen point eight four decibels against seventeen point seven one, and the comparison pits two whole systems with different camera counts, not two decoders. Predicted ego motion alone drops a perfect tracking score to about eighty-four. The faster simulation workflow is measured across software releases and is not isolated. Results are single numbers without error bars.

The idea to carry away is that the structure of the car does much of the work. Factoring each camera into a fixed rig and one moving trajectory, and training every window at the same metric scale, is what lets short windows be assembled into a whole drive's geometry and tracks. Unification itself is useful but not free: one backbone can serve geometry, objects, occupancy, and rendering, yet the tasks still compete, and choosing which ones to train together remains an open question.