AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

AVC-Reward. Event-Centric Audio-Video Consistency Evaluation. By Yuruo Wu and eight others, from Zhejiang University, HiThink Research, and two other institutions. Published on the fifteenth of October. Its code repository is public and real: about twenty source files.

Joint audio-video generation models can now produce scenes with three kinds of complexity: several events composed together, audio mixed in multiple layers, and relations between sound and picture that run asymmetrically. Evaluation has not kept pace. Current metrics are largely reused from earlier tasks that were simple and ran in one direction. They usually judge a clip as a whole, and rely on simple relations defined in advance. For these more complex scenes, that makes them unsuitable. Picture a clip with a door slamming, a dog barking and music underneath. A single score for the whole clip cannot say which of those sounds matched its picture and which did not.

The proposed framework is event-centric. It disentangles audio events from visual events, analyses each event independently, cross-validates relations across the two modalities, and then aggregates the judgments into a single consistency metric with three levels, running from coarse to fine. At the top sits global narrative harmony. In the middle, temporal synchronisation at the level of events. At the finest level, alignment of attributes. The authors present this as a general formulation, meant to hold across diverse generation settings and diverse cross-modal relations. Separate levels let the metric register different failures. A clip can tell a coherent story overall and still have one event out of sync, or have every event in sync and still get a detail of one of them wrong.

Two things are built on the metric. AVC-Reward instantiates it as an audio-video reward model that can be trained. AVC-Benchmark exists to test whether automatic judgments agree with human perception, across different generation paradigms and different degrees of relational complexity. It is built mainly from diverse positive and negative samples, produced by various types of audio-video generation models. Including negative samples matters for the test: an evaluator that agrees with people on good clips has also to agree with them on which clips went wrong.

Among automatic methods, AVC-Reward achieves state-of-the-art alignment with human judgment. It also gives interpretable rationales that localise where a clip fails, rather than a bare score. And used as an online reinforcement-learning signal, it improves the audio-video consistency of a joint generation model.

That last result turns a judge into a teacher. The same instrument that grades a clip event by event can also serve as the training signal for a generator, which means the evaluation and the improvement share one definition of what consistency is.