ActionOPD: Distilling Decisions, Not Tokens

ActionOPD. Distilling Decisions, Not Tokens. By Huining Yuan and seven others, from Tsinghua University. Published on the fourteenth of October. Its code repository is worked out: about four hundred source files, with a readme and a licence.

Distilling a capable language-model agent into a smaller model means aligning the student's behaviour with the teacher's. That raises a basic question: at which states should the alignment happen, and at what granularity? The two usual answers each carry a cost. Offline imitation gives dense supervision, but it is tied to the teacher's exact tokens, and it can suffer from a mismatch between the states seen in training and the states the student actually reaches. Token-level on-policy distillation, or OPD, addresses that mismatch, but it typically needs the teacher's token probabilities and a compatible tokenization. That rules out a teacher you can only query as a black box, or one from a different model family.

The authors argue that the natural unit of supervision for an agent is the semantic action: the decision, rather than the words used to express it. ActionOPD is an action-level on-policy distillation method, motivated by maximizing the expected likelihood of the teacher's actions at the states the student itself encounters.

At each such state, ActionOPD samples actions from both the student and a black-box teacher. Student actions are then rewarded according to how well they agree, semantically, with the sampled teacher actions. Policy optimization does the rest, pulling the student's behaviour at the level of actions toward the teacher's decisions. The student never has to reproduce the reasoning the teacher wrote, and never needs access to the teacher's token probabilities.

Qwen3-4B and Qwen3-8B were trained with two teachers on diverse, curated tool-use tasks. Across four out-of-distribution benchmarks, average gains range from three point three to six point nine percentage points. ActionOPD scales favourably with both student size and teacher capability, and it does so without requiring teacher and student to share a model family or a tokenizer. On tau squared bench, it achieves competitive performance against both offline distillation and token-level OPD.

The authors also explore a variant called Self-ActionOPD, in which the teacher is the same model as the student, given privileged context.

The authors present ActionOPD as a promising approach to agent distillation. What changes is the thing being matched. Supervise tokens, and the student must speak like the teacher, with the teacher's tokenizer. Supervise decisions, and the student only has to act like it, which is why the teacher can stay a black box.