What Outcome RL Buys: Finishing, Reliability, Rarely Reach

What Outcome RL Buys. Finishing, Reliability, Rarely Reach. By Shan Yang, from Individual Researcher. Published on the twenty-sixth of October.

Outcome RL is already known to raise pass at one without raising pass at k when k is large, a result the abstract credits to earlier work by Yue and colleagues. This study starts from a detail that is easy to overlook. Every pass at one is scored under a token budget, and every training recipe also runs under one. When an RL model scores higher, then, what does the gain actually consist of? Is it changed behaviour, or new knowledge? The answer here leans heavily toward behaviour. Gains split into finishing and reliability, while reach, the part that would count as new knowledge, is rare.

The comparison covers twenty released RL models, ranging from one point five billion to thirty-two billion parameters, each set against the thinking model it started from, on physics and math problems at fixed token budgets. Each change in pass at one is divided into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach counts the problems RL solves that its base answered wrongly in all eight tries, minus the problems the base solves that RL answered wrongly in all eight.

Where the base often hits the token budget before answering, most of the change, gain or loss, is finishing: eighty-two percent, at a budget of four thousand tokens and again at eight thousand. Given more tokens, the base closes most of that finishing gap by itself. Where the base has enough tokens, the gain is reliability instead. The RL model is graded right more often, even on problems that both models finish.

Reach, by contrast, adds at most two point one percentage points on any of the four physics and math sets. The authors checked one hundred and forty-four physics problems that only the RL model solved, and of these, three point five percent hold up when resampled or read. In a random sample, nine of ten new RL steps that a human confirmed also appear in the base's own samples. A small-scale RL run on an eight billion parameter model tells the same story: the gain on new problems is finishing, with reliability flat.

The boundaries are drawn carefully. Reach is defined by the authors' own measure, eight tries per problem, and the statement that RL rarely solves what its base cannot holds by that definition. The evidence comes from released models compared at fixed token budgets on four physics and math sets, so the picture belongs to those domains. The authors' own training run is small scale, on a single eight billion parameter model. One further attempt is reported plainly: supplying each problem's physical setup was not shown to improve on plain RL.

Read together, the three words of the title form a ranking. Outcome RL buys finishing when the budget is tight, reliability when it is not, and reach only rarely. A pass at one gain is best heard alongside the token budget it was scored under.