RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

RewardVerse. Rubric-Guided Policy Optimization for Video Reward Modeling. By Zhenchen Tang and eight others, from the Institute of Automation of the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and two other institutions. Published on the nineteenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Reinforcement learning is vital for optimizing video generation models, and a robust reward model is its cornerstone. Existing video reward models often produce unstable scalar scores. They take something complex and subjective, the quality of a video, and map it directly to a single number, with no explicit evaluation criteria in between. The result is what the authors call scalar drift: the scoring scale collapses or shifts from one prompt to another. A reward whose scale moves like that is an unreliable signal for reinforcement learning. A policy trained against it cannot tell whether a higher number means a better video, or only a prompt on which the scale happened to sit higher.

RewardVerse draws its inspiration from professional human annotation engineering. It places a dynamic rubric between the evaluation query and the scorer, as an intermediate representation. Instead of scoring directly and without constraint, the system first generates explicit evaluation criteria, then performs scoring guided by that rubric. The rubric acts as a stable semantic anchor, and the anchor is what mitigates the drift. Think of a grader asked for a single mark with no guidance, and a grader handed a marking scheme first. The second has something to hold on to, and so gives marks that mean the same thing from one question to the next.

That leaves two components that have to work together, a rubric generator and a scorer. To optimize the pair efficiently, the authors propose Rubric-Guided Policy Optimization, or RGPO, a training algorithm in two stages. The first stage warms up the scorer using self-evolving seed rubrics. The second jointly optimizes the rubric generator, so that it produces evaluation criteria adapted to each query, while the scorer is continuously aligned with human ratings. The two pieces are trained to cooperate. The generator learns to write criteria that suit the query in front of it, and the scorer stays tied to what people actually judged.

Extensive experiments use the sixteen-dimensional EvalVerse benchmark along with external datasets. RewardVerse is reported to mitigate scalar drift and to reach state-of-the-art performance on both pointwise and pairwise evaluation. The authors also describe the resulting reward signal as robust and interpretable for reinforcement learning in video generation.

The change is one of order. A score that arrives with no stated criteria has nothing holding it in place from one prompt to the next. Writing the criteria down first, and fitting them to the query, gives each score something to be measured against, and gives a person something to read when they want to know why a video scored as it did.