Learning the 4D World through Point Trajectory Forecasting

Learning the 4D World through Point Trajectory Forecasting. By Byungwoo Jeon and twelve others, from KAIST, ETH Zurich, and four other institutions. Published on the twentieth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

Forecasting what several cameras will see next is only reliable if their videos all describe the same evolving three-dimensional scene. A forecast in which one camera shows an object moving one way and another camera shows it moving differently has failed, however good each video looks alone. The difficulty is that each camera's video is a separate stream of pixels, and nothing in the pixels themselves forces the streams to tell the same story.

The Video-Point Model, VPM, is a world model built to keep the cameras in agreement. It conditions on video histories from multiple views and on a language instruction, and from those it forecasts two things jointly: future videos from arbitrary viewpoints, and the 3D trajectories of persistent points in the scene. The videos are what a viewer would see; the trajectories are a record of how the scene itself moves, independent of any one camera. Putting the second alongside the first is the core of the approach, and the reason the title speaks of a 4D world, three dimensions of space unfolding in time.

It starts from a pretrained world model and extends it in two ways. Camera conditioned multi-view processing integrates the complementary observations coming from different cameras. And a trajectory head predicts how physical points move through time. Each trajectory follows a single point across every view. So when those trajectories are supervised alongside video generation, the appearance changes in every camera are anchored to one shared description of motion in three dimensions.

Two resources support training and evaluation. VPM-Data is a simulation corpus that pairs synchronised multi-view videos with persistent point trajectories. VP-WorldBench is a forecasting benchmark that spans both simulated and real environments.

VPM is also extended to robot control, by adding robot actions to the generator as another token modality. The action tokens share the point-supervised backbone and are denoised together with the future videos. The policy is thereby grounded in the same representation, one that is aware of the 3D scene, rather than in a separate model of its own.

On VP-WorldBench and on existing 4D generation benchmarks, point supervision improves video fidelity and 3D motion accuracy over a counterpart trained on video alone. In simulated and in real-robot experiments, it also raises the manipulation success of the version that outputs actions.

Tracking points is the thread that runs through every part of the system. One trajectory per point, seen by all cameras, is what ties the views together, and the same anchor carries over into the actions a robot takes.