Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

Predicting Privileged Information Effectiveness for On-Policy Self-Distillation. By Kaijian Zou and seven others, from the University of Michigan, Ann Arbor, Mila, and one other institution. Published on the fifteenth of October. Its code repository is worked out: about three hundred source files, with a readme and a licence.

On-policy self-distillation lets a large language model learn from privileged information without a separate teacher model. Prior work has mostly treated that privileged information as a fixed design choice, so how its content and construction affect results is poorly understood. The question is sharper in multi-turn agentic settings, where a trajectory holds actions, observations and reasoning that can be turned into privileged information in many different ways. The paper's central claim is that this choice matters. The effectiveness of privileged information depends on its content, on the source that generated it, and on the student model receiving it, and it can be predicted before training.

The authors systematically evaluate sixteen configurations of privileged information, across three multi-turn agentic benchmarks and three student models that vary in scale and in reasoning mode. Information-dense trajectories generally yield larger gains, and so does privileged information produced by more capable generators. Thinking models benefit less consistently.

Then comes prediction. Two KL-based metrics are compared with the utility of the privileged information itself: sampled-token reverse KL, and error-step KL contrast. The KL-based metrics track post-training performance more consistently than utility does, and the authors note that utility can be distorted by information leakage. Selecting each task's privileged information by the error-step contrast also improves post-training performance over random selection, most of all when the candidates differ widely.

The claims are bounded by the setup: sixteen configurations, three benchmarks, three student models, all in multi-turn agentic settings. Within that, the effects are not uniform. Gains depend on the student, thinking models respond less consistently, and selection by metric helps most when the candidates are far apart. The method is also not presented as a replacement for reinforcement learning. Effective privileged information stays beneficial when combined with GRPO, and the combined objective outperforms a strong GRPO baseline.

There is also a limit on the cheapest predictor. Measuring the utility of privileged information directly looks like the natural test, but the authors find it can mislead, because leaked information inflates it. That is why the recommendation leans on the KL-based metrics instead.

The practical upshot is a change of habit. Rather than fixing privileged information once and leaving it, a practitioner can build several candidates, score them with a KL-based signal before any training, and choose. Content matters, the generator matters, the student matters, and all three can be checked ahead of time rather than discovered after a training run. The authors offer the results as practical guidance for constructing and evaluating privileged information in on-policy self-distillation.