MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception

MinkowskiPE. Minkowski Positional Encoding for Spatiotemporal Perception. By Yuhao Li and six others, from Wave Intelligence Lab, the Chinese University of Hong Kong, and one other institution. Published on the twenty-seventh of September. No code repository is linked to it.

Building physical intelligence, at scales from the microscopic to the macroscopic, depends on modeling how time and space are coupled. Existing models capture that structure in two broad ways. Physics-motivated dynamical formulations give stronger priors, but may constrain flexibility. Learning-motivated architectures are more flexible, but leave the coupling between time and space largely implicit. One side buys structure at the price of freedom, the other buys freedom at the price of structure. The aim here is to have both: flexible learning, plus an explicit geometric bias for modeling time and space jointly.

Minkowski Positional Encoding, or MinkowskiPE, takes that bias from spacetime geometry. It uses joint temporal and spatial coordinates to parameterize Lorentz transformations, and applies those transformations to the query and key features. Time and space enter the encoding together, as one set of coordinates, rather than being encoded separately.

The consequence shows up in the attention score. With MinkowskiPE, the score between a query and a key depends on position only through the relative spacetime displacement between the two tokens. Shift every coordinate by the same amount and the score does not change: it is invariant to global translation. Just as important in practice, the approach keeps the standard dot-product attention interface, and stays compatible with efficient attention implementations. The geometry lives in how queries and keys are transformed, not in a new kind of attention.

Evaluation reaches both ends of the scale range. On microscopic molecular dynamics, MinkowskiPE achieves the best results on all nine multi-trajectory molecular evaluations. On macroscopic video prediction, it reduces mean squared error on KTH by nine point nine percent relative to the best baseline, while using roughly one tenth as many parameters. Molecules at one end and video at the other are the two ends of the range the problem was posed across.

MinkowskiPE puts the geometric bias in the positional encoding and leaves the rest of the model as flexible as before. Time and space are bound together through Lorentz transformations of queries and keys, while attention itself is left alone. The same encoding then serves molecules and video, and because the attention interface is unchanged, efficient attention implementations still apply.