PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

PivotOPD. Learning to Recover from Pivotal Mistakes in Multi-Turn Agents. By Yinghui He and six others, from Princeton University, NVIDIA, and one other institution. Published on the thirtieth of September. No code repository is linked to it.

On-policy distillation is a promising way to train language agents. The student generates its own trajectories, and a teacher supplies dense supervision on them. Multi-turn interaction complicates that picture. An incorrect action changes the states the student will meet later, so one error leads into the next, and errors compound across turns.

The authors looked at how this plays out, in preliminary experiments with three Qwen3 models ranging from eight billion to two hundred and thirty-five billion parameters. More than half of the failed rollouts contained what they call a pivotal mistake: an action that moves the agent farther from completing the task. That mistake typically comes early. And it is often recoverable. Guiding the model for only a few turns after the pivotal turn can be enough to restore task success.

That observation gives PivotOPD its two aims. It is an on-policy distillation framework that trains the student jointly to prevent pivotal mistakes in the first place, and to recover from the states those mistakes create. Prevention and recovery are treated as separate lessons, each with its own signal.

At each pivotal mistake, a teacher model steps in twice. First it provides the gold action, the one the student should have taken. Then, for each of the next few turns, it names a recovery action. Preventive distillation uses the gold action with reverse KL, steering the student away from the pivotal mistake. Recovery distillation uses the recovery actions with forward KL, to transfer recovery behaviours that the student rarely samples on its own. The choice of direction matches the job: one objective pushes away from a known bad step, the other passes on behaviour the student seldom produces.

PivotOPD is tested against thirteen baselines on ALFWorld, WebShop, and Search-based QA. It achieves the strongest average performance for both a Qwen3 one point seven billion student and a Qwen3 eight billion student. On ALFWorld with the smaller student, it improves over the strongest baseline by five point five percent.

The gains also carry over to a different model family and a different domain. In software engineering, PivotOPD raises the resolve rate of a Nemotron three point five student on SWE-Bench Verified by three point two percent.

Here is the episode in one picture. A failed trajectory is often decided at one early turn, and that turn is often not fatal. PivotOPD finds it, shows the student what it should have done, and then walks it through the next few steps from where it actually landed. The student learns the way out, and not only how to avoid falling in.