AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

AdviSD. Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation. By Rishabh Agrawal and four others, from Google and the University of Southern California. Published on the twenty-ninth of September. It links to a code repository that is no longer there.

Picture a small model that cannot touch a large one, but can talk to it. A small trainable advisor steers a frozen language-model executor using advice written in natural language. It can learn from task rewards, and it can also learn from feedback on interactions that have already finished, using that feedback to improve its advice. There is a catch in the second route. A correction can be perfectly plausible and still not change what the executor does. Learning from such a correction nonetheless shifts the advisor's future decisions in other contexts.

The authors make that worry precise. In a model with shared parameters, they prove that corrections of this kind can limit learning, when their targets favor useful advice less strongly than the targets of other corrections. And keeping them less often than the rest improves the model's eventual performance, compared with learning from every correction indiscriminately.

Advisor Self-Distillation, AdviSD, follows directly from that result. It pairs outcome-based reinforcement learning with self-distillation from a copy of the advisor that is conditioned on feedback. The key word is selectively: not every correction is learned from.

Reflection proposes the corrections. To decide which ones to supervise, the advisor scores the same recorded executor response twice, once with the advice it actually issued and once without it. The size of that difference is what selects decisions for supervision. Because the scoring is done by the advisor on a response already recorded, the approach needs neither the executor's likelihoods nor any additional executor rollouts. The large model stays a black box throughout.

The experiments train Qwen3 advisors with eight billion parameters to advise Gemini and Claude. Against advisor-GRPO, AdviSD wins by between four point two and six point four percentage points on BFCL-v3, and by between three point nine and five point one score points on EnvScaler.

The trained advisors also generalize to tasks outside their training domain, and transfer across different executor versions and across model families. A final control matters for the argument: AdviSD beats a random selection that keeps the same number of corrections. That isolates the selection rule itself, rather than simply learning from fewer examples, as the source of the gain.

The logic runs from a proof to a filter. A correction that would not have changed the executor's behavior is not harmless to learn from, since it reshapes the advisor elsewhere. So the method asks a cheap question of each decision: how much does the advisor's own score of the recorded response move when its advice is taken away? That measured difference, and not the mere plausibility of a correction, decides what the advisor is trained on.