Continually Post-training Small Models via Synthetic Self-distillation

Continually Post-training Small Models via Synthetic Self-distillation. By Woosung Koh and six others, from KAIST AI and LG AI Research. Published on the ninth of October. No code repository is linked to it.

Continual post-training has to manage two things at once: acquisition and retention. A model must learn new capabilities while keeping what it already knew. Self-Distillation Fine-Tuning, or SDFT, tackles this with self-supervision built from in-context demonstrations. That approach leans on strong in-context learning and instruction following, and smaller models lack them. As a result SDFT fails on exactly the small models that matter for low-latency, on-device use.

The authors start by locating the root cause. Poor acquisition and forgetting both correlate linearly with how far the post-trained model's distribution shifts. Learning needs to stay in distribution. Synthetic Self-Distillation, called SynthSelf, keeps the in-distribution rewrite that SDFT relies on, but hands that rewrite to someone else: an aligner from the same model family that is only minimally larger.

In SDFT the in-distribution rewrite comes from the model itself, through its own in-context abilities, and those are the abilities a small model lacks. SynthSelf imitates that rewrite with outside help, choosing a helper that is as close to the small model as possible: the same family, and only minimally larger. The idea is help that comes from nearby rather than from far away. The aligner's work is paired with off-policy supervision and with rejection sampling. Together these give the small model the high-quality supervision it needs, without breaking its original distribution.

The evaluation spans four acquisition datasets, one of them a mixture, and fourteen retention benchmarks, at three small model scales. Across them, SynthSelf consistently outperforms six other methods, including both off-policy and on-policy distillation. Further analyses, described as extensive, support the same conclusion.

The diagnosis does the heavy lifting. A linear link between distribution shift and both failure modes turns a vague difficulty into a single quantity to keep small. Once forgetting is tied to distribution shift, the remedy is to keep the training signal close to the model's own distribution, and borrowing a slightly larger sibling to write it is how SynthSelf manages that for models too small to do it themselves.