Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

Rationale-Guided Policy Optimization. Learning to Reason with Adaptive Rationale Scaffolding. By Hoang Phan and five others, from New York University, the University at Buffalo, and one other institution. Published on the fifth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme.

On-policy reinforcement learning has become a central way to improve the reasoning of large language models, and its effectiveness is often limited by reward sparsity. When a model never finds a correct trajectory for a hard problem, optimization gets little useful signal from that problem and may stagnate. A model can only be rewarded for what it manages to produce.

Existing remedies bring in outside help: off-policy demonstrations, expert traces, or solutions generated by models. The catch is format. That auxiliary data typically has to match the format of the reinforcement learning task, and getting it to match often means rejection sampling from stronger models to obtain suitable training trajectories. Ground-truth information may be sitting right there, in a form training cannot directly use.

Rationale-Guided Policy Optimization, or RGPO, uses ground-truth rationales adaptively, according to what the model can currently do, while leaving it free to explore. The key move is in how reference solutions are treated. They are not fixed targets to imitate. They are temporary scaffolds. A scaffold is there while something is being built and comes down once it can stand. How much the model leans on a rationale is meant to follow its current capability, rather than being the same for every problem and every stage of training.

With a rationale in hand, the model generates improved responses. Then only the solutions that are both generated by the model and higher in reward are transferred back to the original setting, the one without guidance. The rationale helps the model reach a better answer, but what gets learned from is the model's own answer, not the reference. Because of that, training can exploit whatever ground-truth information is available, without the off-policy data needing to follow the same format as the reinforcement learning task. The rationale never has to look like a training trajectory. It only has to help the model produce one. And the trajectories it helps produce arrive already in the model's own format, because the model wrote them. That is what removes the need to sample suitable trajectories from a stronger model.

The evaluation covers both language-only and vision-language reasoning. In both, RGPO consistently improves over RLVR baselines. Ablation studies point to adaptive rationale guidance as a key contributor to the gains, which matters because it ties the improvement to the scaffolding itself rather than to incidental changes around it.

The authors present RGPO as a practical and general approach to three linked goals: reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning, in text-only and multimodal models alike. The picture it leaves is of a hint that is offered and then withdrawn. The model gets help to find a better path, and is then asked to walk that path without the help.