Cross-modal Steering Vectors Exist Across Both Image and Text Generation

Cross-modal Steering Vectors Exist Across Both Image and Text Generation. By Donghao Zeng and three others, from Mila and McGill University, Carnegie Mellon University, and two other institutions. Published on the twenty-sixth of October.

Are concepts and mechanisms unified across modalities in current AI models, or are they implemented separately for each one? That is a broad question, and the authors make it testable through cross-modal steering. The test has two steps. First, find vectors that successfully steer generation when added to tokens of one modality. Then add those same vectors to tokens of another modality, with no further adjustment, and see whether they steer that generation too. The claim is that they do. Steering vectors transfer zero-shot, and they do so in both directions: from text to image, and from image to text.

The study runs on several unified multimodal models. These were chosen because they possess shared parameters for language and image generation, and the work focuses on those two modalities. The method matters to the claim. Because steering is fundamentally causal, the test reaches past simple correlational alignment, such as measuring representational similarity. A similarity score says two things look alike inside the model. A steering vector that changes the output shows that the direction does something.

Transfer holds across diverse steered concepts, emotion and object size among them. It holds even for concepts highly unique to one modality. Formal versus colloquial language is a property of words, yet such steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design.

The transfer has a geometric signature as well. A clear modality gap separates text representations from image ones. Despite that gap, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept.

The models studied are unified multimodal ones, picked precisely because their parameters are shared between language and image generation, and the two modalities studied are those. The results describe transfer that often occurs and directions that tend to align, not a guarantee for every concept, every model, or every pair of modalities.

A single direction in a shared model can mean the same thing whether it is added to a text token or to an image token. The authors read this as light on how vision and language interact inside unified multimodal models, and as a pointer towards more general, modality-agnostic ways of controlling their behaviour: methods that need not care which modality they are steering.