EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment

EVA-Cap. Optimizing Audiovisual Video Captioning via Event-Centric Alignment. By Jiapeng Shi and seven others, from Fudan University and OPPO. Published on the fifteenth of October. It links to a code repository that is no longer there.

Omni Language Models offer a unified way to caption video from both its sound and its pictures. Existing approaches, though, mostly treat captioning as unstructured text generation. That neglects the semantic structure a video already has. A video is not one undivided stream of description, and captioning it as if it were has a cost. The cost lands on fine-grained audiovisual alignment, which ends up suboptimal.

EVA-Cap, a new framework, takes that structure seriously. It decomposes captions into atomic audiovisual events, the smallest units in which sound and picture are described together. Supervision that used to be a flat stretch of unstructured text becomes event-centric alignment, with the event as the unit the model is trained against. The caption is still text at the end, but the training no longer treats it as undifferentiated text.

Training runs in two stages, Supervised Fine-Tuning and then Group Relative Policy Optimization, on a dataset the authors curated themselves. It is called EVA-Data, and it was built with event-centric quality control. In the GRPO phase, the model is optimized explicitly against three event-centric alignment targets: global event completeness, intra-event attribution, and inter-event synchronization. One target looks at the events as a whole, one looks inside each event, and one looks across events.

Extensive experiments show EVA-Cap significantly outperforming existing methods on event-centric evaluation, with an average gain of nine percent on ChronusAV. On general audiovisual captioning benchmarks it stays competitive, with a gain of three point three percent on WorldSense.

Treating a caption as a set of events rather than a paragraph is what makes fine-grained alignment something a reward can target directly. The largest gains come on evaluation built around events, while the general benchmarks indicate that the added structure did not cost ordinary captioning quality.