Lluna: Collaborative Unimodal Models for Missing-Modality Inference

Lluna. Collaborative Unimodal Models for Missing-Modality Inference. By Haoyue Bai and five others, from the College of William and Mary, Carnegie Mellon University, and two other institutions. Published on the fourteenth of October. Its code repository is public and real: twelve source files.

Pretrained unimodal models can be made to collaborate, which offers a flexible route to multimodal prediction. The trouble starts when modalities go missing. Then the collaboration can end up less accurate than one of the surviving unimodal models working alone. The authors study this with fixed anchors: a surviving model whose own input does not change. Even so, collaborative gains weaken or reverse as other inputs disappear. Their diagnostics point to two symptoms. Less modality-specific information is recovered after aggregation, and the collaborative representations depend on which inputs are present.

Lluna splits prediction into two routes. One is a direct private path that predicts from adapted unimodal features. The other is a common collaboration branch, built on a graph, whose job is to supply a residual correction on top. The private path gives a prediction directly, and collaboration adjusts that prediction rather than replacing it.

Training adds Progressive Collaboration Consistency. It organises guidance from richer input states to poorer ones, following the removal of a single modality at a time, and it aligns normalised common representations along the way. Task supervision meanwhile adapts predictions to whatever evidence is available.

The evaluation covers eight datasets across affective computing, scene understanding, and medical prediction. Averaged over input states that are not complete, Lluna achieves the best evaluated performance. On the Mean-Missing measure it improves by four point two six percent on average, relative to the strongest evaluated baseline. The fixed-anchor diagnostic also turns positive: under minimal inputs, mean collaboration shows gains on all eight tasks. Code is available.

The starting point is uncomfortable for collaboration in general: with modalities missing, working together can do worse than one model alone. Lluna answers by making collaboration a correction to a private prediction rather than the prediction itself, and by training it from richer states toward poorer ones, one removed modality at a time. Under minimal inputs, the diagnostic that first showed the problem now shows collaboration paying off.