Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

Self-Retrospection Distillation. Turning Post-hoc Experiences into Prior Foresight. By Haoxiang Zhang and eleven others, from Salesforce AI Research, UC San Diego, and two other institutions. Published on the sixth of October.

Reinforcement learning with verifiable rewards turns an agent's experience into something it can learn from mainly through one channel: a scalar outcome reward, handed out after the interaction is over. For group-relative objectives, that channel has a blind spot. When every rollout in a group receives the same reward, all successes or all failures, the signal vanishes. Nothing distinguishes one rollout from another, so nothing is learned. Yet the trajectories themselves were not empty. They may still reveal what the task actually required and how the agent went wrong. That information is sitting in the rollouts and the reward simply cannot see it.

The authors ask a complementary question. Can hindsight teach an agent what it could have anticipated before acting? Their answer is a framing they call prospective learning, which uses experience gathered after the fact to supervise predictions of foresight made from the view before interaction begins. The concrete instance is Self-Retrospection Distillation, or SRD. The intuition is simple to state. A completed trajectory reveals knowledge that would have been useful and pitfalls that should have been avoided. That is privileged hindsight, and SRD distils it into foresight of the same policy, foresight that is blind to the trajectory.

Two details define the arrangement. The teacher and the student are one policy: what it sees in hindsight is used to shape what it predicts in advance. And the foresight exists only as a training target. It does not need to be explicitly generated at inference time, so the deployed agent carries no extra step for it. The method is positioned as a complement to existing training rather than a replacement, sitting alongside reinforcement learning with verifiable rewards and self-distillation baselines.

Across ten tool-integrated reasoning and long-horizon agentic tasks, SRD complements those baselines with gains of up to twenty-four point two percentage points. The advantage is sharpest exactly where the diagnosis pointed, when reward contrast is scarce. Across model scales, between thirty-seven and ninety-eight percent of rollout groups are reward-uniform, and SRD can still draw learning signal from the sampled trajectories.

The starkest case is the two B model scale. There, ninety-eight percent of groups are all failures. Training with verifiable rewards alone ends at zero percent success. Adding SRD, under the same rollout budget, reaches sixty point six percent.

The broader suggestion concerns what experience after the fact is for. It is usually spent on evaluating behaviour or improving it. Here it is also used to shape predictive representations before any interaction is available, so that a group of identical rewards is no longer a group with nothing to teach.