Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching

Beyond Spatio-Temporal Priors. A Generalizable Approach for Dense Correspondence Matching. By Luping Liu and three others, from the University of Hong Kong, ByteDance Seed, and one other institution. Published on the eighth of October.

Dense correspondence matching has long leaned on simplifying assumptions about space and time: motion is smooth, geometry is rigid. For classical tasks those assumptions work. They break in image editing and reference-guided generation, which the authors group as IEG. There, a transformation can keep an object's visual identity while breaking physical continuity entirely, and a matcher that expects smooth motion has nothing to hold on to. The object is clearly the same object to a human eye, but it did not get there in any way that a motion prior would recognise.

FreeMatching aims for correspondence that preserves identity across exactly those transformations. It is a generalizable framework that combines generative and semantic foundation representations. Instead of asking whether a point could have travelled from here to there, it leans on what large pretrained models already represent about appearance and meaning.

Supervision is deliberately mixed. It comes from classical datasets, from tracked videos, and from synthetic scenes. On top of that, teacher-guided iterative refinement improves correspondence in the IEG setting, and does so without any dense correspondence annotations. That matters because labels of that kind are exactly what edited and generated image pairs do not come with.

A single FreeMatching model substantially improves correspondence quality on challenging IEG image pairs, while staying competitive on classical benchmarks. The authors also use it as a quantitative metric for identity preservation, and its scores correlate with human judgment.

Once a matcher stops assuming physics, it becomes useful in a second role. Besides finding correspondences, it can measure whether an edited or generated image kept the identity of its reference. One model, trained on a mix of old and new sources, serves both the classical task and the new evaluation.