Universal Textual Teaching for LLMs

Universal Textual Teaching for LLMs. By Zhanyi Lu and one other, from Westlake University. Published on the eighth of October. No code repository is linked to it.

Knowledge distillation moves what a stronger Teacher model knows into a weaker Student model. Most methods do this by training the Student's parameters, and that choice has a cost. The distilled knowledge ends up bound to one specific architecture and one specific checkpoint. It lives implicitly in the weights, where it is hard to interpret and hard to carry over to any other model. It also means distillation is of limited use for a model you can only reach through an API, or one that is expensive to train. The work studies knowledge transfer for large language models with exactly that limitation in view.

Universal Textual Teaching, or UTT, takes the weights out of the picture. It is a framework that never updates parameters. Instead, it distills the observed knowledge gaps between a Teacher and a Student into a natural-language artifact the authors call a Primer. The Primer is text: something a person can read, something that can be inspected, and something that can be handed to a different model and reused.

Building the Primer happens in two moves. First, UTT identifies representative gap cases, the places where Teacher and Student diverge, through paired evaluations. Then it updates the Primer iteratively through interactions among several roles. The Student attempts each task. A Prompter turns the evaluation feedback into a teaching instruction. The Teacher supplies a targeted demonstration. And a Synthesizer consolidates the lessons that have been validated into the Primer itself. Round by round, the document accumulates what the Student was missing.

The evaluation uses two demanding settings: math, on Omni-MATH-2, and code generation, on KernelBench. On KernelBench, the Student's accuracy rises from nine point four percent to forty-eight point six percent, and its Fast1 accuracy rises from nine percent to thirty-five percent.

On mathematical reasoning, accuracy goes from twenty-seven point six percent to fifty-one point seven percent. UTT is also reported to beat representative prompt engineering methods, and parameter-based distillation methods too. The last result concerns reuse. UTT generalizes across different Teachers and Students: a Primer synthesized for one Teacher and Student pair can carry over to other Students that played no part in building it.

That transfer result is what the textual form was for. A distilled lesson that lives in weights stays with the model that was trained. A distilled lesson written as a Primer is an object of its own, readable by a person and usable by a model that never took part in the teaching, including one whose parameters nobody can touch.