Data Unlearning via Inverse Distillation

Data Unlearning via Inverse Distillation. By Aleksei Leonov and five others, from the AI Foundation lab, MIRAI, and three other institutions. Published on the twenty-eighth of September. No code repository is linked to it.

Multi-step matching models, a family that includes flow and diffusion models, produce high-quality outputs, but they carry two costs. Inference is expensive, because generation takes many steps. And they may reproduce unwanted components of the datasets they were trained on. One problem is about speed and the other about content, and this work sets out to handle both with a single method.

Inverse Distillation Unlearning, or IDU, is a unified framework that does two things at the same time. It distills a multi-step teacher into an efficient student that generates in one step. And it suppresses outputs that correspond to a designated subset of the training data, called the forget set.

The construction starts by formulating distillation as a min-max objective over a data distribution. That distribution is then represented as a mixture of two parts: the forget set, and the distribution the generator produces. The mixture can be compared with the teacher's training distribution, and at the optimum, what is recovered is only the retained data. Put differently, the forget set is written into the comparison itself, so that matching the teacher leaves the student matching only what was kept.

The requirements are light. IDU needs a teacher pretrained on the full data, and data from the forget set. It needs no access to the retained training examples, and no extra feature extractors or classifiers.

Extensive experiments on MNIST and CIFAR-10, under both flow-matching and score-based diffusion settings, show IDU substantially reducing how often the forgotten classes are generated, while preserving generation quality on the classes that are retained.

To the best of the authors' knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models. Its appeal is economy. One framework yields a student that is both fast and has forgotten what it was asked to forget, and the only data it needs beyond the teacher is the data to be forgotten.