Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis

Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis. By Julia Herbst and three others, from Helmholtz Munich, the Technical University of Munich, and two other institutions. Published on the ninth of November.

The Platonic Representation Hypothesis suggests that neural networks are converging toward similar representation spaces. One line of support has been a trend: as language models become more capable, their representations align more with those of vision models. Recent work has questioned whether that trend holds for newer language models. If it stopped, that line of support would go with it. The central claim here is that it persists, on one condition. The captions have to be sufficiently clean and informative. Given captions of that quality, newer and more capable language models still line up more closely with vision models, as the hypothesis expects.

The study proceeds in three steps. It first looks at where alignment between unimodal models emerges, across model layers and across representation dimensions. It then revisits the capability-alignment trend with newer models, which is where the caption condition comes from. Finally, it puts cross-modal alignment in context with unimodal and multimodal baselines on various datasets. Those baselines answer an earlier concern, that similarity metrics can be confounded by model depth and width and need appropriate baselines before they can be interpreted.

Measured against those baselines, the result has two sides. The cross-modal alignment of unimodal models can surpass that of vision-language models trained explicitly to be aligned. Yet it remains substantially behind the alignment observed between models of the same modality. Both statements are relative ones, and each needs its baseline to mean anything: a vision model and a language model can agree more than a model built for both, and still agree far less than two models of one modality do. The persistence of the trend is also conditional on caption quality rather than unconditional.

What the paper supplies is a sense of scale. Vision and language models trained separately do come to resemble each other, enough to rival models built to be aligned, but resemblance across modalities still trails well behind resemblance within one.