TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

TrackEverything. Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations. By Ayush Jain and seven others, from Carnegie Mellon University and Meta. Published on the twenty-fourth of September. No code repository is linked to it.

Point tracking models follow points on surfaces through a video, and existing ones face a fundamental tradeoff. They can track a sparse set of query points over a long horizon. Or they can track every point, but only across short clips. Dense and long, together, has been out of reach.

TrackEverything is a 3D point tracker that sets out to break that trade-off. It rests on one insight: a video is a set of 2D projections of an underlying 3D world. So instead of tracking in the image, it represents the video as persistent 3D scene tracks in world coordinates. That changes what the cost depends on. Model complexity is decoupled from how long the video runs, and scales instead with how much unique physical scene geometry there is. A camera that keeps looking at the same room does not keep adding new things to track.

Three innovations make it work. The first is de-duplication. At the boundaries between sliding windows, a voxelization-based mechanism merges tracks that sit in the same place, so repeated observations of the same surface do not pile up redundantly.

The second is a split in the tracking itself. An endpoint refiner predicts where each point ends up, and classifies it as static or dynamic. A lightweight trajectory refiner then decodes dense trajectories only for the dynamic points. Points classified as static need no trajectory decoded at all, so the heavier work is confined to the part of the scene that is actually moving.

The third is 3D WAFT. Correlation volumes in four dimensions are prohibitive in memory, and 3D WAFT replaces them with efficient feature sampling in the scene cloud. Each of the three innovations attacks cost from a different side: de-duplication limits how many tracks exist, the refiner split limits how many of them need full trajectories, and 3D WAFT limits what each one costs in memory.

To the authors' knowledge, TrackEverything is the first 3D tracker that can track all visible points across videos longer than one thousand frames, within forty gigabytes of GPU memory. On TAPVid-3D, it outperforms every open-source all-frame dense 3D tracker by more than twenty percent APD on short clips. On long sequences, it stays competitive with state-of-the-art sparse trackers, even though it is tracking far more points than they are.

Three decisions carry the approach. Let cost follow the scene rather than the running time. Merge what has already been seen. And spend the detailed trajectory decoding only on what moves. Together they are how a tracker aims at all points and long videos at once.