Harnessing Textual Refusal Directions for Multimodal Safety

Harnessing Textual Refusal Directions for Multimodal Safety. By Moreno D'Incà and two others, from the University of Trento. Published on the sixteenth of October. Its code repository is public and real: about twenty-five source files.

Making a multimodal large language model safe usually depends on unsafe multimodal data. That data is used either for post-training alignment or for activation steering. The trouble is the data itself. It is hard to collect. It has to be gathered anew for every modality, so images need their own set and video needs another. And collecting and distributing it raises ethical and legal concerns. The safety recipe is bottlenecked on material nobody really wants to hold.

The authors relax that constraint by asking a narrower question. A language model backbone already carries refusal directions, extracted directly from text. Do those textual directions generalize across modalities, to image and to video? Preliminary findings say yes, with conditions. How well it works depends on which layer is chosen, how strongly the model is steered, and on cross-modal alignment. That last condition has a specific failure: it causes safe multimodal inputs to be spuriously steered toward refusal.

Modality-Agnostic Refusal Steering, or MARS, is built on those findings. It is lightweight and training-free, and it injects multimodal safety without any multimodal safety data. Each of the three conditions gets its own answer. Modality misalignment is corrected through activation centering. Steering strength is scaled adaptively, inside a geometrically defined trust region. And the method selects the optimal intervention layer. All of this operates at the first generated token.

MARS is evaluated on five state-of-the-art multimodal models, across safety, utility, and video jailbreak. It achieves consistent safety gains while preserving utility, and the abstract states that preservation with a qualifier: up to over-refusal. Spurious refusal of harmless inputs was the failure the preliminary findings flagged, so that is the boundary to keep in mind when reading the result.

The broader reading the authors draw is about where safety lives. MARS reveals, in their words, that safety-relevant structure is shared across modalities. If that holds, the refusal behaviour already present in the text backbone is a foundation to build on, and they call textual refusal directions powerful and underexplored as a basis for multimodal alignment. The practical payoff is that the hard, sensitive part of the pipeline, unsafe multimodal data, is no longer required at all.