ReLift: Dense 3D Tracking as Lifted Trajectory Repair

ReLift. Dense 3D Tracking as Lifted Trajectory Repair. By Assaf Singer and four others, from the Technion, Israel Institute of Technology and NVIDIA. Published on the thirtieth of October.

Dense 3D tracking from monocular video asks for three things at once: point correspondences over time, the depth of those points, and their trajectories while they are hidden. Existing methods usually learn all three jointly, which is a demanding objective when 4D supervision is scarce.

The authors start from an observation. Strong pretrained 2D trackers and 3D geometry models already provide an effective foundation. Lift the 2D tracks into 3D, simply interpolate the occluded stretches, and the resulting trajectories already rival dedicated 3D trackers that use the same geometry. They are not clean, though. Lifted trajectories inherit the correspondence errors and depth errors of their sources, and under occlusion their depth comes from the occluder rather than from the point being tracked.

ReLift is an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. Existing components produce the trajectories, and what remains to be learned is narrower: keep what is right, fix what is corrupted, and fill in what is missing. The model never has to find correspondences or estimate depth from pixels. It receives trajectories that are already mostly right, and its task is to make them right everywhere, including the stretches where the point could not be seen.

At its core is a trajectory corrector that alternates attention along trajectories and across them. It has three jobs: preserve the reliable observations, repair corrupted positions, and complete the missing motion.

The inputs are purely geometric, and that choice pays off in training. The model can learn directly from 3D trajectories with simulated lifting errors, which decouples training from images, and from any specific tracker or geometry model. All of the scarce 4D supervision is concentrated on trajectory refinement. The result is a reusable model that consistently improves lifted trajectories across trackers and geometry providers, without retraining, and so it benefits directly from future advances in either component.

On WorldTrack, ReLift achieves state-of-the-art mean accuracy. It improves on the strongest baselines by two point one APD on all points, and by three point zero on dynamic points. Its faster configurations remain competitive with leading 3D trackers while running faster than 4RC. The larger of the two gains falls on the dynamic points, the subset that is moving, and the speed comparison means the repair step does not have to come at the cost of a slow pipeline.

Rather than learning tracking from scratch, ReLift learns only the correction. Correspondence and depth come from models trained elsewhere, and the scarce supervision is spent where those models go wrong: on corrupted positions, and on the stretches hidden behind an occluder. Because the corrector sees only geometry, it is not tied to the tracker or the depth model it was trained alongside. Swap either one for something better, and the same repair step still applies.