Distillation Defenses Easily Break After Reinforcement Learning
Shidan Javaheri, Alexander Panfilov, Oliver Britton, Yarin Gal, Yonatan Gideoni
University of Oxford · ELLIS Institute Tübingen · MPI for Intelligent Systems
Published September 28, 2026 · on the feed September 29, 2026 · ▲ 3 · arXiv 2609.35699
The abstract
Distillation attacks copy the reasoning capabilities of closed-source large language models, allowing bad actors to replicate state-of-the-art performance at low cost. Attackers systematically collect a large volume of frontier model reasoning traces and then train (i.e., "distill") their own models on these traces. Existing defenses against distillation attacks are typically evaluated immediately after distillation, implicitly assuming attackers do not train their models any further. In this paper, we argue that a more realistic threat model includes further training with reinforcement learning after distillation. A misspecified threat model can give a false sense of security -- some defenses that seem effective after distillation can be broken after subsequent reinforcement learning. Practically, reinforcement learning lowers the bar for a distillation attack to be effective. We show that simple attacks can steal reasoning capabilities from existing closed-source language models using data easily obtainable from current APIs, yielding reasoning improvements equivalent to more sophisticated attacks that extract the full hidden traces. Results indicate that any distillation defense that leaks sufficient information to reconstruct approximate reasoning traces is likely ineffective. We conclude by discussing broader implications and batch-level distillation defenses which could be more effective.
The Brief
2:08A short podcast summary of the paper. Written and voiced by AI.
Transcript
Distillation Defenses Easily Break After Reinforcement Learning. By Shidan Javaheri and four others, from the University of Oxford, the ELLIS Institute Tübingen, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about three hundred source files, with a readme.
A distillation attack copies the reasoning ability of a closed-source large language model. The attacker collects a large volume of reasoning traces from a frontier model and trains their own model on them, reproducing state-of-the-art performance at low cost. Defenses against this are typically evaluated right after distillation, which quietly assumes the attacker stops training there. The claim of this work is that a realistic threat model has to include what comes next: further training with reinforcement learning. Under that threat model, some defenses that look effective after distillation break.
The central demonstration is about how low reinforcement learning sets the bar. Simple attacks, using data that is easy to obtain from current interfaces, can steal reasoning capability from existing closed-source models. Once reinforcement learning follows, the reasoning gains are equivalent to those of more sophisticated attacks that extract the full hidden traces. In other words, hiding the traces buys less than it appeared to, because the later training stage recovers much of what was withheld.
A misspecified threat model is not only an academic worry. The paper describes it as giving a false sense of security: a defense passes the evaluation it was tested under and then fails against an attacker who simply keeps going.
The general conclusion is stated with care. Any distillation defense that leaks enough information to reconstruct approximate reasoning traces is likely to be ineffective. That is a condition, not a verdict on every defense. It points at what a defense would need to withhold. The authors close by discussing broader implications, and batch-level distillation defenses, which they suggest could be more effective. The abstract leaves those as a direction rather than a tested result.
The lesson is about where evaluation stops. A defense measured at the moment distillation ends is measured against an attacker who does not exist. Extend the attacker by one ordinary training stage, and the protection that looked solid has to be judged again.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A defense that passes its test is not the same as a defense that works. Start there.
You guard a closed-source reasoning model. Attackers pose as normal users, harvest its reasoning traces in bulk, and train their own cheap models on them.
The road ahead
- 1 · The problem ~7 min Defenses against copying a reasoning model are tested at one moment, and that moment may not be where a serious attacker stops.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Attackers likely keep training with reinforcement learning after copying traces, and judged there, trace-hiding defenses can stop working.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The attack pipeline collects traces, distils a student on them, adds reinforcement learning, and only then scores the defense.
- 6 · The experiment ~6 min You’ll guess: With reinforcement learning after distillation, how did simple attacks using easily obtained API data compare with attacks that extract the full hidden reasoning traces?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.