HRoPE: Homography Relative Positional Encoding

HRoPE. Homography Relative Positional Encoding. By Huachen Gao and eight others, from Zhejiang University, Westlake University, and three other institutions. Published on the twentieth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Multi-view transformers look at a scene from several cameras at once, and the main way of giving them spatial inductive bias has become geometric relative positional encoding. Current designs condition attention on camera-level frustums or on ray-marched depth. Either way, tokens from different views are associated through the geometry of the sensor that captured them. The authors' objection is that this sensor-centric framing only describes where a token is observed from. It leaves out the surface that the token actually depicts, and the geometry of that surface is rich information the encoding never uses.

They argue for a shift toward positional encoding that is surface-aware, one that explicitly grounds each patch token in the geometry of a physical surface. HRoPE does this by representing the local geometry of every patch as a slanted plane in three dimensions. The choice is not new to this paper in spirit: it follows the slanted support-window assumption of classical multi-view stereo, carried over into a transformer.

A plane seen from two viewpoints induces a homography between them, and those homographies are what HRoPE puts to work. It injects token-level, plane-induced homographies into attention as relative transformations. That equips the transformer with a geometric prior for two jobs: rectifying perspective distortions, and aligning patch features across viewpoints. Where earlier encodings related two tokens through the cameras that saw them, HRoPE relates them through the surface they both depict, so the relation attention receives is a statement about the scene rather than about the sensor. The encoding stays relative, as positional encodings of this family are, but what it is relative to has changed.

The encoding does not require ground-truth geometry. The plane parameters can be predicted end to end, trained with a view-synthesis objective alone, or they can be derived from foundation models. Either source plugs into the same encoding.

Extensive experiments cover three datasets: RealEstate10K, CO3Dv2 and DL3DV. Across them, the best variant of HRoPE consistently outperforms the geometric positional encodings the authors evaluated, on the task of novel view synthesis. Extensive ablations and analysis are reported alongside, to show that the proposed encoding is what produces the effect.

The move is a change of reference. Instead of telling attention where the camera stood, HRoPE tells it what plane the patch lies on, borrowing its assumption from classical multi-view stereo and needing no measured geometry to do so.