HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction

HiP3R. Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction. By Zunnan Xu and three others, from S-Lab at Nanyang Technological University, Imperial College London, and one other institution. Published on the twenty-second of October. Its code repository is worked out: about ninety-five source files, with a readme and a licence.

Feed-forward models estimate camera motion and 3D geometry accurately from short video windows. On sequences of thousands of frames, though, computational and memory costs rule out direct inference, so long videos must be processed as overlapping chunks. That division leaves behind independent local reconstructions, each in its own inconsistent coordinate frame, along with duplicate estimates of the cameras the chunks share. Global consistency needs constraints carried across the whole sequence, both from adjacent overlaps and from distant revisits. That requirement points naturally toward pose graph optimization.

The difficulty is choosing the level of the graph. Chunk-level graphs efficiently resolve scale drift and close large loops, but their rigid transformations cannot correct errors inside a chunk, or conflicting predictions for the same frame. Frame-level optimization offers exactly that local flexibility, but scaling it globally is computationally expensive, and it can break metric scale. Each level fails where the other succeeds. HiP3R resolves the trade-off with a hierarchical pose graph that uses both, in sequence.

The first step optimizes a robust chunk-level Sim three graph, which establishes the global layout and the scale. The second step holds that layout fixed and estimates one shared SE three camera per frame. That is what turns the overlapping window predictions into a single trajectory. Where several chunks each produced their own estimate of the same frame, there is now one camera for it, placed inside a layout that has already been settled.

Experiments across multiple benchmarks show HiP3R improving both camera trajectories and 3D reconstruction. The abstract reports no figures. It does stress how broadly the method applies: it works with both chunk-based and streaming backends, and it needs no additional training or fine-tuning, so it sits on top of an existing model rather than replacing it.

The design splits the problem along the line where each graph level breaks. The coarse graph handles what rigid chunks are good at, global layout, scale and large loops, and is then frozen. The fine stage only has to reconcile frames inside that frozen layout, which is how it can be flexible per frame without paying the cost, or risking the scale breakage, of optimizing every frame globally.