TraceForge: Distilling with Your Own Thinking

TraceForge. Distilling with Your Own Thinking. By Yan Wang, working independently. Published on the twenty-first of October. No code repository is linked to it.

Using one LLM to generate training data for another is among the most general ways to train a model, and it is still common practice. Some frontier teams have closed that route for their own models. Ask one of them and what comes back is an answer, or a short summary, without the detailed reasoning trace that produced it. The author finds this a little ironic: those models were themselves built by distilling human knowledge, distilling an open-source model remains entirely routine, and the restriction applies only to the providers' own outputs.

Recent work has recovered the hidden reasoning by replaying encrypted blocks across models. The abstract grants that it somehow works, but notes that it breaks the providers' terms of service, which makes it nothing a product can rest on for the long term.

TraceForge is the alternative. It takes the context and the teacher's answer as input, and writes the reasoning trace itself, in the style of the student's own model family. The teacher supplies the answer. The student's family supplies the voice the reasoning is written in.

Training has two stages, supervised first and then adversarial. A generator learns to regenerate the family's own reasoning traces, given their context and answer. A critic learns how real a reasoning trace looks. Once trained, the generator is applied to a teacher's answer, and writes a trace for it in the family's style.

The trick in the training data is that the family's own traces come complete. For those, the context, the answer, and the real reasoning between them are all available, so the generator can be taught to recover the reasoning from the two ends. The critic adds the adversarial half, learning to tell how real a trace looks, and it can also be used to pick among several candidate traces. At use time only the two ends are needed, which is exactly what a closed teacher still returns: the context you sent and the answer it gave.

An independent judge rates how likely each trace is to be genuine, on a scale from zero to one. Generated traces average zero point six eight. Real traces average zero point six six. Traces chosen by the critic from four candidates average zero point eight two. The abstract is explicit that these are authenticity scores, not rates of correctness. By that judge's reading, the generated traces look about as genuine as real ones, and the critic's picks look more genuine still.

The downstream test uses five benchmarks, with everything else held fixed. On general tasks, the five training corpora come out about the same. Coding is where they separate: there, the generated reasoning beats Fable 5's own thinking content. With the same five thousand questions and the same teacher answers, HumanEval+ improves by three point seven points and MBPP+ by seven point one at the final checkpoint, and the MBPP+ lead holds at every checkpoint along the way.

The method changes only who wrote the reasoning. The questions are the same and the teacher's answers are the same, but the trace leading to each answer is generated in the student family's own style. On coding, that was enough to beat training on Fable 5's own thinking content. The author offers an explanation, flagged as a suspicion rather than a result: reasoning written in the student's family style may simply be easier for the student to learn from.