Solv3R: 3D Looped Transformer as a Multi-view Solver

Solv3R. 3D Looped Transformer as a Multi-view Solver. By Jeong-gi Kwak and four others, from the University of British Columbia, Wayve Labs, and four other institutions. Published on the second of November.

Classical structure-from-motion reconstruction works iteratively. It refines camera poses and scene geometry starting from a current estimate. That habit has two practical benefits. A map can grow without recomputing views that were already computed, and the geometry can improve as more observations arrive. Most geometric foundation models give that up. They either predict the entire geometry in one forward pass, or update a recurrent state as frames come in one after another, and in both cases expanding or refining a reconstruction becomes non-trivial.

Solv3R brings the iterative habit back inside a learned model. It is a looped transformer that learns to continue solving multi-view geometry from whatever state it is currently in. Its update operator does not depend on the iteration count. That matters because it lets views that have already been refined and views that were just introduced be updated together, regardless of how far along each one is.

The operator is trained for two behaviours at once. It should preserve a reconstruction that is already accurate when it keeps iterating, and it should correct a state that is inaccurate. Together those allow a fixed collection of images to be refined, and new observations to be folded in. There is an efficiency gain as well. The representations of views that are solved can be cached once and reused as geometric context across iterations, which makes key-value caching compact and efficient.

The authors analyze the solver's dynamics and compare it with the state of the art. The demonstration is that a single learned solver supports offline reconstruction, incremental reconstruction, relocalization and streaming, with no task-specific training and no task-specific modules.

Each of those settings leans on a property described earlier. Offline reconstruction is the refinement of a fixed collection of images. Incremental reconstruction is the integration of new observations into a map that already exists, which is where an operator indifferent to the iteration count earns its keep, because old and new views are updated side by side. Cached representations of solved views let earlier work be reused rather than recomputed as the solver keeps going.

Four settings that could each have had their own training and their own modules are served by one learned solver. That follows from asking the network to keep solving rather than to answer once: a solver that can pick up from any state does not need to know whether the state came from a fresh start, a growing map or a stream.