Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation. By Yilei Tu and four others, from the University of British Columbia, the Shanghai Artificial Intelligence Laboratory, and three other institutions. Published on the twelfth of September. Its code repository is worked out: about sixty-five source files, with a readme.

Specialist distillation passes domain expertise to a student model through reasoning trajectories that the teacher generates. The student learns by imitating how the specialist reasons. But suppose the specialist itself was trained only on question and answer pairs, with no explicit reasoning supervision at all. Nothing told it how to reason, only what to answer. What, then, governs the trajectories it produces? The claim is that specialist optimization implicitly selects from a latent space of possible trajectories, and that the tuning choices made during that optimization decide what gets passed downstream.

To observe that latent distribution, the authors use student distillation as a probe rather than as the goal. The logic is that a student inherits no parameterization and no optimization constraints from the specialist. It inherits only the sampled trajectories themselves, so whatever the student becomes reflects what those trajectories carried, and nothing else.

Through this probe a tight relationship appears. Across twenty-seven specialist and student pairings, the specialization and generalization profiles of teacher and student correlate exceptionally strongly. Then comes the controlled test. Explicitly controlling the specialist's distributional drift shifts both the teacher and its distilled student, systematically, along a trade-off between domain precision and retention of general capability. Turn the dial on the teacher, and the student moves with it.

The evidence comes from chemistry, physics and multilingual settings, where distilled students reflect the specialist-induced profiles even across divergent model families. The claim is scoped to one condition: specialists trained without gold reasoning. When reasoning supervision exists, it is not what is being studied. And the relationship is shown through the probe: what is observed directly is how students behave, which is then read back as evidence about the trajectories they were given.

The practical reading is that, absent gold reasoning, there is no neutral way to train a specialist. Every tuning choice made on the teacher is also a choice about the implicit supervision a future student will learn from, whether or not anyone intended it that way.