Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning. By Heming Zou and nine others, from Tsinghua University and Tencent. Published on the sixteenth of October. No code repository is linked to it.

Reinforcement learning with verifiable rewards has become a scalable way to post-train large language models. Training agents over long interactions, though, raises two problems. With only a terminal reward, credit for each turn is sparse and high-variance. And when several mini-batch updates run in sequence on one fixed batch of rollouts, the current policy drifts away from the behaviour policy that produced them, so the data grows stale. The first problem is about where the learning signal comes from. The second is about how far each update is allowed to go.

Existing fixes each carry a cost. Learned critics, process reward models and on-policy distillation give denser credit, but they need a value network, step-level annotations, or a stronger external teacher, all of which add training complexity. Importance sampling and clipping mitigate the drift, but they do not specify a finite target that would stop sequential updates from overshooting. They slow the drift without saying where it should end.

TreeOPSD addresses both problems at once. It is a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Credit comes from the structure of the rollouts themselves rather than from a learned critic, and the update is aimed at a teacher distribution with a built-in limit.

Each interaction turn is treated as a node in a tree. TreeOPSD branches alternative continuations from shared prefixes, and recursively backs up the terminal rewards into implicit turn-level credit. That credit is implicit: it comes out of the tree itself, with no value network and no step-level annotation. Each credit is then mapped to a closed-form, bounded teacher distribution, parameterised by the likelihood ratio relative to the frozen behaviour policy. Training minimises the divergence between that teacher and the student policy. Two properties follow. At the behaviour policy, the direction of that update is the policy-gradient direction, recovered as the tangent. And across sequential mini-batches, the updates are bounded toward a finite target instead of overshooting.

Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially on long-horizon tasks. In the Deep Research setting, for example, it raises the average accuracy of Qwen3-30B-A3B-Thinking-2507 by four point nine seven points over GRPO. That emphasis on long horizons matches the setting the method was built for, where both credit assignment and policy staleness were framed as the obstacles.

One closed-form object does double duty here. The bounded teacher turns tree credit into a per-turn learning signal, so no critic or annotator is needed, and its bound gives sequential updates somewhere to stop. Credit assignment and policy staleness, usually handled by separate machinery, are answered by the same target.