On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

On-Policy Self-Training. Data-Free Self-Improvement in Large Language Models. By Leon Sick and five others, from Google and Ulm University. Published on the twenty-third of October. No code repository is linked to it.

On-policy self-distillation lets a foundation model be post-trained without a stronger external teacher. Existing approaches still depend on supervision of another kind, though: ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. Removing the teacher took away one dependency. The curated questions and answers are a second one, and they remained in place.

This work reformulates on-policy self-distillation as a self-training framework. The question-answer pairs are generated by the model itself, and so are the privileged reasoning demonstrations. Supervision is pushed to its limit. The only guidance is a coarse, high-level description of a topic. From that alone, the model synthesizes its own curriculum of queries and of privileged reasoning traces, without access to any human-curated prompts or answers.

Self-distillation then runs on that fully synthetic privileged information. The model writes the curriculum and learns from it, with the topic description as the only outside input. The authors apply the same framework to text and to multimodal domains, to show that it is general rather than tied to one kind of input.

They find that self-distillation on fully synthetic privileged information lets a model improve across multiple domains, while retaining its capabilities on domains it is not self-trained on. The self-improvement is consistent on standard language benchmarks and on Visual Question Answering. Both halves of the finding matter. Improvement on the self-trained domains shows that the synthetic curriculum carries real signal. Retention on the other domains shows that the gain is not bought by losing what the model could already do.

A coarse topic description is the whole of the input: no curated prompts, no reference answers, no stronger teacher. From that thin starting point the model still improves, and keeps what it could already do elsewhere. The authors conclude that foundation models possess rich latent capabilities, and that those capabilities can be amplified without external task supervision or curated problem distributions.