Inducing Process Supervision from Outcome-Only Reinforcement Learning

Inducing Process Supervision from Outcome-Only Reinforcement Learning. By Shengda Fan and four others, from Renmin University of China, Tsinghua University, and two other institutions. Published on the twenty-ninth of September. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

Process reward models give a language model feedback one step at a time, and that step-level feedback supports both post-training and reasoning at test time. Strong ones are costly to train. Human annotation of individual steps is difficult to scale. Monte Carlo estimation, the usual alternative, is computationally expensive, and it can drift away from whether a step is actually correct. Either way, the step labels themselves become the bottleneck: expensive to get, and not always faithful when you do get them.

TIPS, short for Thinking-Induced Process Supervision, sidesteps the step labels entirely. It is an outcome-only reinforcement learning framework for training generative process reward models. Nothing in training tells the model whether any individual step was right. The wager is that step-level judgment can be induced from a reward that only ever looks at the end.

For each solution it judges, the model writes a chain of thought, then a label for each step, then a label for the outcome. The reward depends solely on whether that predicted outcome matches the ground truth. The group-relative advantage computed from this reward is then used to optimise the entire generated response, step labels included. Why should that teach anything about steps? When checking the intermediate steps helps decide the outcome, more accurate checks lead to better outcome judgments, and better outcome judgments earn higher rewards. Reinforcing the final verdict therefore reinforces step-level verification along with it, without explicit process supervision ever being supplied.

The authors validate TIPS across math and agent benchmarks and across four backbone families. The headline result is a model built on Qwen3-4B-Thinking-2507. Trained with only three point two thousand outcome-labelled trajectories, it reaches eighty-five point two F1 on ProcessBench. That surpasses every trained process reward model in the evaluation, and it also surpasses strong prompt-only judges such as GPT-5.4-Instruct and Claude-4.7-Opus. It does still trail o1-mini. Code and data are publicly available.

The trick here is one of economy. A label that is cheap to collect, whether the final answer was right, ends up training a skill that normally needs the expensive label, whether each step was right. The model is never shown what a good step looks like. It learns to check steps because checking them is what gets the outcome right.