Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

Adaptive Reward Routing. Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL. By Songlin Yang and seven others, from the Hong Kong University of Science and Technology, Tencent Video, and one other institution. Published on the twenty-ninth of September. No code repository is linked to it.

A joint audio-video diffusion model has to get several things right at once. The picture has to be good and the sound has to be good, each on its own terms. The two have to agree in meaning. And they have to line up in time. Reinforcement learning guided by multiple rewards offers a promising way to improve such a model along all of these complementary objectives together.

How well that works depends on two quantities, and both change as training proceeds. The first is where the reward-driven updates should act inside the model. The second is how competing rewards should be combined. Existing methods tend to rely on fixed routing and fixed reward weights, set once. A fixed choice cannot follow a moving target, and so these methods fail to track how the model's functions evolve over training.

Adaptive Reward Routing jointly adapts both: where updates land, and how the rewards are coordinated. It does this during forward-process RL, specifically DiffusionNFT, applied to joint audio-video diffusion models. The method has two components, one answering each of those questions.

Cross-Modal Influence-Guided Routing localizes the updates. It takes the bidirectional cross-attention responses as an efficient proxy for how cross-modal influence is evolving. Guided by that proxy, it dynamically reweights token-aware losses and scales gradients across the cross-modal layers. None of this requires additional interventions in the model.

Preference-Preserving Modality-Aware Reweighting coordinates the rewards. It keeps the predefined reward weights, treating them as preference priors rather than discarding them. After a warm-up period, it adds residual corrections drawn from branch-specific interactions between reward gradients. That lets it resolve conflicts as they evolve, without letting dominant rewards suppress objectives that are weak but essential.

Extensive experiments show consistent improvements over strong RL baselines, on all three fronts the method targets: modality quality, semantic consistency, and audio-video synchronization. Ablations and mechanism analyses further support the design choice of having two components. Adaptive update routing and reward coordination each bring something, and the benefits are complementary rather than overlapping.

Fixed weights and fixed routing treat the training problem as if it stood still. Adaptive Reward Routing treats both as moving targets, reading cross-attention to decide where to push and gradient interactions to decide how the rewards balance, while the stated preferences stay in place as the starting point.