Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents
Mingju Chen, Can Lv, Jinrong Liu, Huan Zhang, Heng Chang, Shiji Zhou
Beihang University · Tsinghua University
Published September 27, 2026 · on the feed September 29, 2026 · ▲ 4 · arXiv 2609.33391
The abstract
Reinforcement learning with verifiable rewards (RLVR) often relies on sparse outcome rewards, providing coarse supervision for long-horizon agents. On-policy self-distillation (OPSD) complements this signal with dense privileged feedback. However, we identify Decision--Timestamp Mismatch: privileged guidance may be misaligned with the student's functional decision because the corresponding decision can occur at a different timestep, while the student's decision itself may span multiple timesteps rather than being tied to a single timestamp. Thus, timestamp-local supervision can misalign both the context and the temporal scope of credit. To address this mismatch, we introduce AlignOPSD, following the principle of aligning supervision before assigning credit. Decision-Aligned Supervision Rectification re-scores the same student-sampled response in functionally matched contexts across sibling rollouts to calibrate local teacher evidence. Semi-Markov Hierarchical Credit Assignment then derives variable-duration decision spans from correspondence changes and uses rectified evidence to allocate outcome-grounded credit across spans and their constituent turns. We evaluate AlignOPSD with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA against representative baselines. AlignOPSD outperforms both GRPO and StepOPSD across all eight backbone--aggregate-metric comparisons, improving on GRPO by 5.5--8.7 \% and ranking first in six. Additional analyzes examine the two alignment stages and hyperparameter sensitivity between tasks. Our code is avaliable at https://github.com/mingju-c/Align-OPSD
The Brief
2:34A short podcast summary of the paper. Written and voiced by AI.
Transcript
Beyond Timestamps. Decision-Aligned On-Policy Distillation for Long-Horizon Agents. By Mingju Chen and five others, from Beihang University and Tsinghua University. Published on the twenty-seventh of September. Its code repository is worked out: about four hundred source files, with a readme and a licence.
Reinforcement learning with verifiable rewards, RLVR, often relies on sparse outcome rewards, and for long-horizon agents that gives only coarse supervision. On-policy self-distillation, OPSD, adds dense privileged feedback on top. The authors identify a flaw in how that feedback lands, which they call a Mismatch between Decision and Timestamp. Privileged guidance may not line up with the decision the student is actually making, because the corresponding decision can happen at a different timestep. And a student's decision may stretch over several timesteps, rather than belonging to one timestamp. Supervision that is local to a timestamp can therefore get both the context and the temporal scope of credit wrong.
AlignOPSD is built on a principle stated in one line: align supervision before assigning credit. First make sure the teacher's evidence refers to the right decision. Only then decide how much credit each part of the trajectory deserves.
Two stages carry this out. Decision-Aligned Supervision Rectification takes a response the student sampled and re-scores that same response in functionally matched contexts across sibling rollouts, calibrating the local teacher evidence. Semi-Markov Hierarchical Credit Assignment then derives decision spans of variable duration from changes in correspondence. It uses the rectified evidence to allocate credit grounded in outcomes, across those spans and across the turns inside them. Decisions become units of varying length, rather than single steps on a clock.
AlignOPSD is evaluated with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA, against representative baselines. It outperforms both GRPO and StepOPSD in all eight comparisons across backbone and aggregate metric, improves on GRPO by between five point five and eight point seven percent, and ranks first in six of the eight. Further analyses examine the two alignment stages separately and the sensitivity to hyperparameters across tasks.
Dense feedback helps only if it is attached to the right moment. This work argues that the right moment is a decision, and that decisions do not map neatly onto timestamps. Rectify the evidence first, then spread credit over spans of the right length, and the distillation signal meets the student where its choices were actually made.
Paper Quest · 6 chapters · about 38 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A long task, one reward at the end, and a teacher whose advice keeps landing on the wrong turn.
You are training a language agent on long tasks like online shopping, and the only reward arrives at the very end. It cannot tell a good decision from a lucky one inside a long trajectory.
The road ahead
- 1 · The problem ~7 min A single end-of-task reward is too coarse for long agent tasks, and turn-by-turn teacher feedback can land on the wrong moment.
- 2 · The key idea ~9 min Align the teacher's evidence with the student's actual decision first, and only then decide how much credit each decision and turn deserves.
- 3 · Choose your own research ~3 min
- 4 · Build the method ~8 min Sample sibling attempts, re-score each student response in matched contexts, cut decision spans where matches shift, then share the outcome reward across spans and turns.
- 5 · The experiment ~6 min You’ll guess: Across eight comparisons of backbone and aggregate metric, how often did AlignOPSD beat both GRPO and StepOPSD on long-horizon agent tasks?
- 6 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.