ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

ReSAIL. Mitigating Collapse in Iterative Agent Self-Distillation. By Shengjie Jin and four others, from Renmin University of China. Published on the thirtieth of September. Its code repository is worked out: about three hundred source files, with a readme and a licence.

Iterative self-distillation lets LLM agents learn from successive deployments, which offers a path toward recursive self-improvement. The authors' experiments with existing methods show a collapse instead. Deployment performance falls from cycle to cycle, and task performance with privileged information declines along with it. Both measures falling together means the decline is not confined to the student acting alone: the behaviour that serves as supervision is degrading as well. Since each cycle's student becomes the next cycle's teacher, whatever is lost in one cycle is missing from the supervision of the next.

The response has two parts: prioritize the interaction steps that are most informative for distillation, and preserve PI-conditioned behaviour, meaning behaviour given privileged information, as the student becomes the next teacher. ReSAIL, short for Retentive and Selective Augmentation for Iterative Self-Distillation, packages both as a plug-in augmentation for iterative self-distillation based on privileged information. The two words in the name map onto the two parts of the response, and plug-in means it is added to an existing method rather than replacing one.

The selective part picks the interaction steps where privileged information most strongly changes the teacher's predictions, and balances the resulting distillation losses across trajectories. The retentive part regularizes the student's PI-conditioned output distributions toward those of the frozen teacher. It does this at both selected and unselected steps, so that the behaviour needed to supervise the next cycle is still there when the student takes over as teacher. Selection decides what the student learns from. Regularization decides what it keeps.

On ALFWorld and TextCraft, ReSAIL sustains substantial gains across model scales over three cycles. Added to self-distillation baselines, it brings an average absolute gain of twenty-two point five percent in final-cycle success rates. That gain is measured where the collapse was, at the final cycle, after the student has become the teacher more than once. Separately, sensitivity-guided selection of offline data improves action prediction accuracy for multimodal GUI agents on AITZ.

The authors present this as the first evidence that a more robust learning mechanism can effectively mitigate performance collapse in iterative agent self-distillation over deployment trajectories. Read through the method, the collapse is a succession problem. A student that loses its privileged behaviour cannot hand it on, so the fix is to keep that behaviour alive while learning from the steps that matter most.