SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

SyncReward. Human-Aligned Synchronization Reward for Audio-Visual Generation. By Kaisi Guan and six others, from Renmin University of China and Tencent WeChat. Published on the twelfth of October. Its code repository is public and real: sixteen source files.

Joint audio-visual generation has progressed rapidly, but synchronization is still a challenge: generated speech and sound often fall out of step with the visual events they belong to. A key bottleneck is that the tools for judging this are missing. There is a lack of human-annotated data, of reliable evaluation benchmarks, and of synchronization-aware evaluation models for generated audio-visual outputs. The authors respond with a study spanning all three: data, benchmarking, and reward modeling.

The data comes first. SyncReward-Data-80K contains eighty thousand professional synchronization ratings, for clips from eight audio-visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. What is being measured is human judgment of whether sound and picture line up, gathered at scale and across very different kinds of sound.

For evaluation, SyncReward-Bench is a held-out benchmark of one thousand clips, each rated independently by multiple professionals. Keeping it held out separates the clips used to test a judge from the ratings used to build one, and having several professionals rate every clip gives each a judgment that does not rest on one person. With it, the authors systematically evaluate two families of existing judges: specialized synchronization metrics, and omni-modal models. The question put to each is simple. When people say a clip is in sync or out of sync, does the automatic judge agree?

The first finding is a negative one. Those metrics and models show substantial misalignment with human judgments. The authors then build SyncReward, a reward model that combines semantic-temporal contrastive pretraining on real clips with human supervision on generated clips.

As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the methods evaluated. As a reward, it improves synchronization in two ways: through best-of-N selection at inference time, and through post-training based on reinforcement learning.

The three pieces build on each other. Professional ratings make a trustworthy benchmark possible. The benchmark shows that existing judges disagree with people. And the same human signal trains a judge that can serve as a reward, closing the loop from measurement back to generation. The data, benchmark, model and code are all slated for release.