To Align or Not To Align: Check Your COMPASS Before You Train

To Align or Not To Align. Check Your COMPASS Before You Train. By Fabian Gröger and two others, from EPFL, the University of Basel, and one other institution. Published on the thirteenth of October. Its code repository is public but slight: eight source files.

A common way to build an AI system is to combine pretrained models, whether by aligning a vision encoder with a language model, or by merging fine-tuned checkpoints. Finding a pair that works together usually means a costly empirical search, because whether two models are compatible is known only after the alignment or the merge has been done. Every bad pairing is discovered the expensive way, by paying for the training that reveals it. With many candidate encoders and checkpoints, the number of pairs to try grows quickly, and so does the bill.

COMPASS, the COMPatibility ASsessment Score, is a diagnostic with theoretical motivation that estimates whether a model pair will be compatible before any aligning or merging happens. It builds on an observation about representational convergence: it is most robust at the level of local neighborhoods. So compatibility is measured there, and split into two axes that are cheap to compute.

The first axis is pair-specific ease, defined as the k-NN overlap between the two encoders: how much their nearest-neighbor structure already agrees. The second is a paradigm-level ceiling, the maximum overlap that the two models' training paradigms allow. One axis describes this particular pair as it stands. The other describes how far any pair from those two paradigms could ever get. Keeping them apart is what lets a low score be read correctly: a pair can be far from agreeing yet have plenty of room to improve, or it can be capped by its paradigms no matter how much effort goes in.

Together, the two axes rank candidate pairs and also say what to do with each one. Pairs that already agree, with compatible paradigms, go first. Pairs with compatible paradigms but low initial agreement deserve additional compute. When a paradigm cannot support enough agreement, switch the pretraining paradigm. And pairs that score poorly on both axes are deprioritized.

COMPASS is applied to seventy-two cross-modal alignment experiments and to three hundred and ninety-two model merges done by task arithmetic. Its compatibility estimates correlate strongly with downstream success, with an average Pearson correlation of zero point nine one on the cross-modal alignment tasks, and zero point seven on model merging. The same score, in other words, carries across two quite different ways of combining models, with the stronger agreement on the alignment side.

The value of a check like this is in the order of operations. Compatibility used to be something you discovered by paying for the training. Here it is read off the neighborhoods the two models already have, before a single step is taken, and turned into a decision about where to spend.