Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

Ask to Distill. Learning to Synthesize Data for On-Policy Distillation. By Jiashuo Sun and eleven others, from the University of Illinois Urbana-Champaign, the Hong Kong University of Science and Technology, and one other institution. Published on the twentieth of October. Its code repository is worked out: about one thousand source files, with a readme and a licence.

On-policy distillation can succeed with surprisingly few training questions. That observation invites a view of the questions themselves as carriers of distributional knowledge transfer: what gets moved from teacher to student depends on what the student is asked. Which raises the question the paper takes on. How do you judge whether a newly generated question is valuable for distillation, and learn to generate better ones, when you do not have its correct answer?

Ask-to-Distill treats synthetic questions as distribution elicitors. A question is good if it exposes useful teacher supervision along the trajectories the student itself generates, and that is what the framework optimizes for. The generator, called a synthesizer, is initialized from the student and then trained against a distributional surrogate for question value.

The surrogate is what makes the unsupervised setting workable. It is computed from a frozen student and teacher pair, with no reference answers, no correctness rewards, and no per-candidate student updates. Its reward combines two signals: disagreement between teacher and student, and how much teacher probability mass sits on the student's candidate support. Controls for length and diversity are folded in alongside those. The questions that come out are then filtered and deduplicated into a prompt-only bank, which feeds standard on-policy distillation.

Tests span mathematical reasoning, code generation, and agentic tasks. Against unoptimized student-generated prompts and against KL-filtered real questions, the method improves average performance and teacher-gap recovery. In the settings evaluated it matches or exceeds on-policy distillation run on a bank of real questions.

Signal analyses add a mechanism-level reading. Optimized questions can elicit denser supervision and more total signal from shorter student responses, which the authors offer as support for elicited distributional supervision as an objective for synthetic distillation data.

Notice what is never needed: an answer. The value of a question is read off the gap between two frozen models and where the teacher's probability lands on what the student was already going to say. Ask the right thing and the teacher reveals more per token, which is a different way of thinking about data for distillation. The code, data and models are publicly available.