MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation. By Aryan Das and two others, from the Indian Institute of Technology Kanpur and Amazon. Published on the thirtieth of October.

Self-distillation of large language models often uses a teacher that is an exponential moving average of the student. That teacher already keeps a history of the student's updates. Muon keeps one anyway: a separate momentum buffer, which adds to the memory that has to persist through training. Two records of the same past are being stored.

Muon with Reused Teacher, or Muon-RT, drops the duplicate. For the hidden weight matrices it removes the momentum buffer and reuses the displacement between teacher and student in its place. The authors show that this displacement encodes an exponentially weighted history of the updates that were applied, and that under a local model it smooths out noise that is uncorrelated over time.

The displacement is processed in three steps before it is used. Muon-RT removes its radial component, normalizes it spectrally, and then mixes it with the current gradient after Newton-Schulz orthogonalization. On the theory side, under explicit regularity and alignment assumptions, the authors prove linear contraction of an expected objective-gap bound toward a residual error level.

Under Self-Distillation Fine-Tuning on Qwen2.5-7B-Instruct, Muon-RT raises average accuracy over AdamW by two point six points while using forty-two percent less persistent memory. Against momentum Muon, it gains five point four points with twenty-seven percent less persistent memory.

A second setting, Self-Distillation Policy Optimization on Olmo-3-7B-Instruct, shows an average accuracy gain over AdamW of two point zero points.

Here the memory saving and the accuracy gain come from the same move. The teacher was always going to be stored, and the paper's argument is that its distance from the student already behaves like the momentum Muon would otherwise keep separately. Reusing it frees the buffer, and the results show accuracy going up rather than down.