Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Scal3R. Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction. By Chin-Yang Lin and seven others, from National Yang Ming Chiao Tung University and NVIDIA. Published on the third of September. No code repository is linked to it.

Online 3D reconstruction models perform poorly on long videos, and the reason is structural rather than a matter of capacity. These systems regress camera poses relative to a fixed first-frame anchor. The longer the video runs, the further the current frame sits from that anchor, and the further the model is forced to extrapolate beyond anything it saw in training. Small drifts do not stay small. They accumulate, they amplify, and eventually the geometry collapses.

The move this work makes starts from an observation about how that collapse actually looks. Per-frame depth remains stable throughout the failure. The backbone's local geometry stays intact, and only the global pose head breaks down. The model has not lost its grip on the scene. One component has lost its grip on where the camera is. That decoupling is what the method is built on: if the anchor is the problem, stop relying on a single one.

Scal3R reformulates online reconstruction as multi-reference relative pose querying. Poses are queried relative to multiple past keyframes rather than one fixed frame at the beginning. The mechanism is deliberately small. Lightweight learnable tokens, making up about one percent of the parameters, are injected into a completely frozen backbone through asymmetric attention. The backbone that was working is not retrained at all. On top of that sits an online pose-graph optimization system with loop closure, which suppresses the long-range drift that accumulates over a long sequence.

Training converges in eight hours on a single GPU. On KITTI, average ATE drops by more than sixty percent against the online baseline, and the approach reaches state-of-the-art performance across Virtual KITTI, Sintel, TUM-Dynamic, ScanNet and 7-Scenes.

The interesting part is the diagnosis rather than the architecture. A failure that looked like whole-model breakdown turned out to be one head extrapolating, and the fix was to freeze everything that was working and change only what the broken part was being asked to do.

Scal3R. Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction. By Chin-Yang Lin and seven others, from National Yang Ming Chiao Tung University and NVIDIA. Published on the third of September. This is the long version: the whole paper rather than the abstract.

Here is the failure this paper is about. You point a camera at the world and walk. A feed-forward reconstruction model builds geometry as you go, frame by frame, in real time. For the first stretch it is excellent. Then, somewhere out along a trajectory spanning hundreds of meters, the reconstruction does not degrade gracefully. It collapses. The geometry folds up.

To see why, you need one detail about how these models work. Systems like CUT3R and STream3R decode each frame's geometry into a single global coordinate system, and they do it by regressing the camera's pose relative to the very first frame. Everything is anchored to frame one. Those two systems get there differently, and the paper is careful about this because its fix has to work for both. CUT3R is a persistent state model: each frame is processed together with a hidden state carrying the scene history, which the frame then updates. STream3R is a causal transformer, aligning each frame against a sliding window of recent features instead. Different machinery, same decoding structure underneath. Both maintain image feature tokens for local geometry and one dedicated camera token, and both regress the global pose from that camera token. So both inherit the same weakness, which is the global-anchor paradigm, and two things about it are fragile. The training data covers limited scene scales, so a model trained on short sequences is being asked to extrapolate to coordinates far outside anything it saw. And for an unbounded video stream, the global coordinate system will eventually meet trajectories that are simply out of distribution. Small feature drift does not stay small when it is being amplified through an increasingly distant reference.

That decoupling is the whole design. If local geometry is reliable and the global mapping is not, then stop asking for the global mapping. Scal3R reformulates the problem as multi-reference relative pose querying: rather than regressing an absolute pose against frame one, it queries the pose relative to several recently maintained reference frames, which keeps every question inside the local viewpoint domain where the model is trustworthy. The backbone is frozen completely. What gets added is a small set of learnable tokens, about one percent of the parameter count, injected by a mechanism the authors call asymmetric attention. The asymmetry is the important part, and it repays being precise about. Pose tokens attend to image features as queries. Image tokens perform self-attention only among themselves. Information flows one way, so the pose machinery can read the geometry without disturbing the representation space that produces it, and the point cloud coming out of the frozen point head is unchanged. A shared base query token acts as a template, and for each reference slot the stored camera token of that keyframe is projected through a lightweight MLP and added to it, which is how one learned template becomes a query about a specific past viewpoint. Behind that sits an online backend: pose-graph optimization aggregates the relative constraints into a globally consistent trajectory, with loop closure to catch revisits.

On KITTI, average trajectory error drops by more than sixty percent against the online baseline, and the method reaches state-of-the-art performance across Virtual KITTI, Sintel, TUM-Dynamic, ScanNet and 7-Scenes. The breadth matters as much as the number, because these span both driving-scale outdoor sequences and indoor scenes, and a fix aimed only at long-range drift would have no reason to help on the indoor ones. It also helps to place the method against its neighbours. Adapting frozen 3D foundation models to new tasks without retraining the backbone is an established line of work, and prompt tuning on a frozen CUT3R had already been demonstrated for joint human-scene reconstruction. What is new here is applying that paradigm to relative pose estimation, recasting pose as a multi-reference query rather than something regressed, while the asymmetric injection preserves the pointmap quality the backbone already had.

The authors name two limits, both structural rather than incidental. Performance is bounded by the frozen backbone, so wherever it fails, under occlusion or on textureless surfaces, Scal3R fails with it. Freezing is what makes the method cheap and it is also what caps it. And the online backend has weaknesses of its own: appearance-based loop closure can miss a revisit under extreme viewpoint or illumination change, and both keyframe selection and loop detection lean on hand-set thresholds.

The transferable idea is the diagnosis, not the architecture. Something that looked like whole-model breakdown turned out to be one head extrapolating past its training distribution, and the evidence for that was a plot showing two error curves diverging. Once you know which component broke, the fix stops being a bigger model and becomes a smaller question asked in a place where the model already knows the answer.