MInTRL: Off-policy Intervention can boost On-policy RL

MInTRL. Off-policy Intervention can boost On-policy RL. By Mingyu Chen and four others, from Amazon Web Services and Boston University. Published on the eleventh of September. No code repository is linked to it.

Reinforcement learning with verifiable rewards is typically performed on- policy. That keeps the training data close to the current policy, but it also limits learning to the trajectories the policy can discover by itself. Off- policy methods, supervised fine-tuning among them, can draw on external knowledge beyond what the base model is capable of, yet they may suffer from large distribution shift. One side is learnable but confined. The other reaches further but drifts. The key challenge, as the authors put it, is to expand exploration without sacrificing learnability. Put simply, the method has to reach answers the policy would not find alone, without handing it data so foreign that it cannot learn from it.

Minimal Intervention Reinforcement Learning, or MInTRL, expands the exploration frontier through sparse, local interventions placed inside rollouts that are otherwise on-policy. The policy still writes almost everything. Only occasionally, and only briefly, does something else step in. The corrections are short, they arrive at chosen points, and the rest of the rollout belongs to the policy that is being trained.

During generation, a judge-intervention policy periodically reviews what the current policy has produced. Where the output has gone wrong, it replaces the erroneous suffix with a short correction, and then immediately returns control to the policy, which carries on from there. During training, MInTRL adopts a sequence-level advantage-regression objective, and that objective eliminates the need for importance sampling. The two halves fit together. Generation keeps the trajectories close to the policy, and the training objective is built so that the small off-policy pieces do not require reweighting.

The authors show that these sparse, local interventions can substantially improve coverage beyond what finite-budget on-policy sampling reaches, while the resulting trajectories keep their overall on-policy nature. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off- policy baselines.

The ablations add three more observations. MInTRL remains effective with self- intervention, where the policy intervenes on its own output. It remains effective across different judge policies. And performance peaks at moderate intervention intensity rather than at the heaviest, which is the authors' case for intervening minimally.

The picture these results support is that on-policy and off-policy need not be an either-or. Keep the trajectory the policy's own, let a judge correct it at a few points, and exploration widens without the data drifting away from the learner. The authors present minimal intervention as an effective paradigm for enhancing on-policy RL.