On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training

On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training. By Shufan Shen and seven others, from the Institute of Computing Technology, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the twenty-ninth of September. Its code repository is worked out: about four hundred and fifty source files, with a readme.

On-policy post-training is known for generalizing well, and that has prompted studies of how it updates a model's parameters. Those studies, the authors argue, treat what they observe as a byproduct of on-policy training and stop there. They overlook the possibility that those behaviors could serve as principles in their own right. So this work asks a different question. Is there a specific on-policy update behavior that could be lifted out and used as an optimization principle for a different paradigm, supervised fine-tuning, or SFT, to make it generalize better?

The answer they arrive at is yes, and the behavior is direction. The generalization advantage of on-policy training can be transferred to SFT through the direction in which parameters are updated. Once that direction has been identified, even SFT generalizes well, provided its updates are constrained to follow it.

The path to that claim starts with an analysis comparing the two paradigms. SFT updates parameters along consistent directions. The on-policy paradigm keeps adjusting its direction as training goes on. One method settles on a heading and holds it; the other keeps revising its heading. That contrast pointed the authors toward one quantity in particular, as a promising candidate: the cumulative update direction of each parameter, meaning where that parameter has been moved in total over the course of training.

To test whether that quantity is what carries generalization, they built a method around it. On-Policy direction-constrained Supervised Fine-Tuning, or OPSFT, takes the direction identified by on-policy training and constrains SFT updates to it. OPSFT performs strongly, and that performance is the evidence for the claim. If constraining direction alone lets SFT inherit the generalization, direction is doing the work.

The claim is about where the advantage lives, and it comes with a condition attached. SFT generalizes when its updates are held to a direction that on-policy training has already identified. Some on-policy training is still needed to find that direction. The abstract does not frame this as replacing on-policy methods, but as combining their generalization with what SFT is good at.

It names two such strengths. The first is efficiency. Directions that support strong generalization can be identified using only a few on-policy training steps, and OPSFT then carries training forward at the lower cost of SFT. The second is the ability to use high-quality trajectories. A model that has already been post-trained can keep improving on such trajectories with OPSFT, moving along its existing update direction without disrupting what on-policy training taught it. In both cases the on-policy phase supplies the direction, and the supervised phase supplies the speed or the data.

A simple image holds the result. On-policy training keeps turning as it goes, and supervised fine-tuning walks in a steady line. Aim that steady line the way on-policy training was heading, and the generalization advantage comes along with it, while SFT keeps its own efficiency and its use of good trajectories.