VASPO: Self-Distillation with Privileged Value Estimation

VASPO. Self-Distillation with Privileged Value Estimation. By Fei Wu and three others, from the University of Science and Technology of China and iFLYTEK Research. Published on the fourteenth of October. Its code repository is worked out: about four hundred and fifty source files, with a readme and a licence.

On-policy self-distillation gives reasoning models dense supervision through a privileged self-teacher. But that supervision reflects local token preferences. It does not reflect the expected return of the student continuing on its own, unguided. Recent methods pair this signal with GRPO, letting GRPO set the direction of each update while the teacher scales the signal. The authors argue the integration belongs somewhere else, and that it works better inside PPO.

The argument rests on two signals that are strong in different places along a response. Privileged information makes a critic's value predictions more accurate toward the end of the response. Self-teacher guidance follows a different curve: strong at the start, weaker in the middle of generation, and strong again at the finish. Because the two are complementary by position, the method uses both. Value-Augmented Self-Distillation Policy Optimization, or VASPO, has a self-teacher and a critic evaluate the same privileged information at prefixes the student generated.

The roles are divided cleanly. The critic predicts continuation returns and computes token-level PPO advantages from them. The self-teacher then modulates selected positive advantages, through probability ratios that are bounded. The critic sets the advantage, and the teacher adjusts some of the positive ones within limits.

Evaluation spans six mathematical reasoning benchmarks. Over PPO, VASPO raises mean accuracy by two point five nine percentage points on a Qwen3-4B variant. On an OLMo3-7B variant the gain is one point eight one points. On Knights and Knaves, it reaches the highest mean accuracy among the methods evaluated. The code is public.

Underneath the method sits an observation about timing. A critic with privileged information is most trustworthy late in a response, and a self-teacher is most informative early and again at the end. Rather than choosing one, VASPO gives each its own job inside PPO, so the dense signal of self-distillation is tied back to an estimate of where the student's own continuation will actually lead.