Function-Space Temporal Self-Distillation for Policy Optimization

Function-Space Temporal Self-Distillation for Policy Optimization. By Aryan Das and two others, from the Indian Institute of Technology Kanpur. Published on the fourteenth of October. No code repository is linked to it.

Reinforcement learning with verifiable rewards, or RLVR, improves reasoning in language models, but its supervision is coarse: a single scalar outcome is spread over a whole response. Self-distillation offers denser guidance at the level of tokens, by comparing the policy working unassisted with a self-teacher that has been given feedback. The catch is that this teacher evolves along with the student. Its targets keep moving, which makes them unstable and calls for extra regularization to hold things together.

Function-Space Policy Optimization, or FSPO, keeps a single policy and combines two things: adaptation using privileged information in the present, and consolidation over time in function space. The temporal half works by storing sparse historical next-token distributions and treating them as fixed targets. Those stored distributions supply the separation in time that a stable teacher would, without a persistent teacher model having to be kept around.

Three further mechanisms sit on top. Privileged disagreement correction explicitly suppresses actions that are locally overvalued. Reliability-adaptive grounding strengthens reference supervision whenever self-generated evidence keeps failing. And a temporal memory that is aware of both quality and coverage retains behaviours that have been verified, and improves the privileged demonstrations.

Across reasoning and tool-use benchmarks, under a training budget of five hours, FSPO has the highest average accuracy, about four percentage points above the strongest baseline on average. It does so while bringing peak memory down from about seventy gigabytes to fifty-six on Olmo3-7B-Instruct.

The bet in FSPO is that a teacher does not need to be a model at all. A sparse record of what the policy used to predict can serve as the fixed target, and on the one model where memory is reported, that came with lower peak memory and higher accuracy together rather than one traded for the other. Whether the same holds at other model sizes is an open question the abstract leaves standing.