Pumpire: Unified Benchmark for Metric Distance Estimation

Pumpire. Unified Benchmark for Metric Distance Estimation. By Siyu Chen and eight others, from Zhejiang University, the Zhejiang University of Technology, and one other institution. Published on the eighth of October.

How well does a 3D foundation model know how far apart two points really are? Previous evaluation does not answer that directly. It usually scores depth and camera intrinsics separately, or compares point clouds with geometric similarity metrics. Neither one reflects a model's ability to estimate the distance from one point to another. The underlying ability, perceiving physical scale in reconstructed 3D space, has been largely overlooked.

Pumpire measures metric point-pair distance estimation, directly from the reconstructed geometry. Pick two points in a scene, ask how far apart they are in physical units, and compare the model's answer with the true one. That single question folds together everything a model must get right to know scale, rather than scoring the ingredients one at a time. It covers both image-level and video-level 3D foundation models, with or without depth priors.

The benchmark rests on a large and diverse dataset called pumpire-6k. It holds one hundred real-world scenes. Each scene contains sixty-four frames, and each is annotated with point-pair distances that were physically measured. In total that is six thousand four hundred frames.

On top of the dataset sits a holistic evaluation protocol. It handles image and video models alike, assesses point-pair distance errors directly, and allows comparisons across settings.

The authors run extensive experiments across twenty-nine baseline configurations of representative 3D foundation models, and provide a comprehensive analysis of the results. Those configurations span the range the protocol was built for: models that work on single images and models that work on video, some given depth priors and some not. Because every one of them is scored on the same physically measured distances, they can be placed side by side, which earlier protocols scoring depth or point clouds in isolation did not allow. The abstract does not report headline numbers from that analysis, so the contribution to take away here is the instrument itself, and the settings it makes comparable.

A point cloud can look right and still be the wrong size. Pumpire checks the measurement rather than the look: distances measured physically between real points, compared against what the model reconstructs. By targeting that more fundamental ability, perceiving physical scale, the benchmark asks whether a reconstruction is right in physical size and not merely right in shape.