SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

SpecMRL. Redesign Matryoshka Representation Learning within Parametric Spectrum. By Zhongxin Chen and five others, from Renmin University of China, Lenovo Group, and one other institution. Published on the nineteenth of October. Its code repository is public and real: about twenty-five source files.

Matryoshka Representation Learning, or MRL, is designed so that one embedding can be used at many sizes. It jointly optimizes nested prefixes: the first few dimensions should work as a small embedding, a longer prefix as a larger one, and so on, like dolls inside dolls. The design is effective, but it lacks an explicit constraint ensuring that the leading portion really is a suitable compact encoding. Training asks each prefix to work, without arranging the space so that the front is where the most useful information sits.

SpecMRL reorganizes the feature space so that reading out a prefix becomes more effective. It extends MRL with a spectral basis that is derived from the model's parameters and organized by feature importance.

Beyond that, SpecMRL follows standard MRL. The change is where the prefix is read: readout happens in the learned spectral space rather than in the original one. Because that basis is ordered by importance, cutting it short keeps the directions it ranks first.

On MTEB, SpecMRL outperforms MRL at every reduced dimension evaluated, and its gains grow larger as compression becomes more aggressive. It also extends the performance frontier of MRL. Using only half the embedding dimensions, SpecMRL surpasses MRL at its full dimensionality.

The authors read these results as a sign that the learned spectral basis better prioritizes informative directions, which makes representations more compressible. The pattern of gains fits that reading. If the ordering is what helps, it should matter most when only the first few dimensions survive, and that is exactly where the margin is widest.