EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving

EmoHarness. Boosting Multimodal Emotion Understanding with Meta-Harness Evolving. By Fenghai Li and six others, from the University of Illinois Urbana-Champaign, the University of California, Davis, and one other institution. Published on the ninth of November.

Small multimodal models can get emotion wrong when the signals disagree. The words say one thing, the voice says another, and the face suggests a third. Faced with that conflict, a small model misjudges what the person is feeling. The usual remedy would be to change the model. This work leaves the model exactly as it is.

EmoHarness keeps the small model frozen and calls it the student. What changes is the harness around it: the prompts, the settings, and the verification code that wrap each of its calls. Everything the student sees and returns passes through that harness, so improving the harness is a way of improving the student's answers without touching its weights.

The starting point is a human-written harness called the scaffold. It numbers the student's visual observations, the transcript, and descriptions of prosody, turning each one into a piece of citable evidence. Then it has the student weigh candidate emotions against that evidence.

On top of the scaffold sits Meta-Harness Evolving, or MHE. A stronger language model, the Builder, reads the student's errors and execution traces offline and edits the harness. Static checks and validation gates decide which of those edits are kept. Throughout, the Builder never sees the evaluation items, which keeps the editing apart from the test.

Evaluation uses five thousand five hundred and seven items from MME-Emotion. Against a single video-only call, EmoHarness raises task-macro Recognition by seven point six to thirteen point four points. That holds for all four students and for both Builders.

The breakdown is more careful than the headline. Transcripts and prosody descriptions supply most of the boost, between five point three eight and ten point seven one points. The scaffold, applied to the audio-augmented call, shifts results by anywhere from minus zero point seven nine to plus two point five four, partly by producing extra multi-label predictions. Evolved edits add at most zero point nine five, and no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, giving small gains for three of the four students, though none of those survives correction either.

Read against its own numbers, the gain lives mostly in the inputs. The authors conclude that EmoHarness boosts small multimodal students chiefly by supplying audio-derived evidence and organizing it, while the evolved edits contribute little that holds up statistically.