OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation

OPD Before RL. Warm-Starting Rubric-Based RL with On-Policy Distillation. By Xinpeng Wang and five others, from New York University and Meta. Published on the second of October. No code repository is linked to it.

A great many useful tasks for language models have no exact answer to check against. Rubric-based reinforcement learning is one response to that: an open-ended response is scored against explicit criteria, and the score becomes the reward. The weakness is timing. The reward arrives only once the response is complete, so the training signal says how the whole answer did without identifying which individual decisions along the way earned or lost the score.

The proposal is to use the rubric twice, in two different roles. First it serves as privileged context for a teacher, which turns it into dense, token-level supervision. Only afterwards does it become a reward for further reinforcement learning. The same criteria that were too coarse as a single end-of-response score can, in the hands of a teacher who can read them, guide the student one token at a time.

The first stage is called rubric-privileged on-policy distillation, or RP-OPD. The student never sees the rubric. A teacher that does see it supplies next-token distributions, and the student learns to match them at prefixes the student itself generated. Distillation of this kind eventually levels off, and the authors observe that plateau. The second stage takes over from there: reinforcement learning optimizes the rubric reward directly and carries the model beyond where distillation stopped.

The framework is evaluated on health and science tasks with open-weight models, across HealthBench, ResearchQA and RubricHub Science. The comparison spans several post-training methods, and it varies how much supervised fine-tuning or RP-OPD training happens before reinforcement learning begins. Among the methods evaluated, the two-stage framework achieves the highest scores.

The more telling contrast is about reward hacking on RubricHub Science. RP-OPD followed by reinforcement learning shows only limited signs of it. The baseline that runs supervised fine-tuning before reinforcement learning increasingly collects high rewards for responses that claim to comply with the rubric without actually providing the content the rubric asks for.

Order is the argument here. Rubric rewards alone leave a model guessing which choices mattered, and a model warmed up with plain fine-tuning learned, in this comparison, to talk its way to the reward. Letting the rubric teach first, through a teacher that can see it, gave reinforcement learning a starting point it exploited less. The authors read their findings as support for distilling with rubrics before rewarding with them.