Source Identification Is Not Fitness Testing: Measuring the Limits of Synthetic-Data Attribution

Source Identification Is Not Fitness Testing. Measuring the Limits of Synthetic-Data Attribution. By Joss Armstrong, from Ericsson. Published on the thirtieth of September. No code repository is linked to it.

Training repeatedly on data generated by models can degrade the models that come later. One possible defence is provenance: knowing where each generated example came from, and using that to decide which examples to reuse. This study tests two things about that idea. How reliably can provenance be recovered at all, and does knowing it help pick better training data? The claim it arrives at is in the title. Identifying where data came from and identifying which data are useful for training are separate problems.

The test material is financial-risk text. First, the source of generated passages is identified, and then the same test is repeated after the passages have been rewritten. On the original passages, attributing a passage to its generator is ninety-eight point seven percent accurate. After paraphrasing, that falls to fifty-three point one percent. After style rewriting, it falls to twenty-nine percent. Telling generated text from human text, by contrast, stays close to perfect against the human comparison set that was tested.

The second experiment runs three rounds of generation and retraining, and compares two rules for selecting generated examples. One rule uses source information. The other uses a score from a separate reference model. The two rules select different examples. But the planned comparison does not detect a stable difference in how much the resulting models degrade.

The limits are stated plainly. The text is from one domain, financial risk, and the near-perfect detection of generated text holds against the specific human set that was tested. The selection comparison covers three rounds and two rules, and finds no stable difference rather than a proven equivalence. The conclusion is deliberately narrow: neither the provenance score nor the tested proxy facing the selection criterion is established as sufficient to predict future recursive behaviour.

The design pulls apart three things that are easy to run together: the identity of a source, the selection made against a criterion, and the outcome of recursive training. Attribution proved fragile under rewriting, and even where it works, knowing a passage's origin did not settle whether it was good to train on.