Exploring The Empirical Privacy Landscape of Synthetic Data

Exploring The Empirical Privacy Landscape of Synthetic Data. By Zihao Zhao and five others, from Johns Hopkins University and OpenAI. Published on the fifteenth of October. No code repository is linked to it.

Training a language model on sensitive data forces a trade between what the model gains in capability and the risk that it leaks private information. Methods differ in where they strike that balance. The claim of this study is that generating synthetic data from the sensitive records strikes a more favourable balance than intervening in training itself, particularly when the rewriting is privacy-aware: guided by an instruction that attends to uniqueness and replaces private details.

The comparison is controlled. Data-level and training-level interventions are set side by side on synthetic medical records, which contain canaries planted to probe for leakage. Direct finetuning performs best on capability metrics, but leaks the most. DP-SGD reduces leakage, at a substantial cost to capability. Generating synthetic data from the records offers the more favourable balance of the three.

Compared with naive paraphrasing, privacy-aware rewriting improves privacy protection, and it does so across a variety of different rewriting models. The naive version simply restates a record in other words. The privacy-aware version is told to look for what makes a record unique and replace those private details. More capable rewriting models also generally reduce the risk of downstream leakage, so the choice of rewriter matters as well as the instruction it is given.

Several qualifications come with that, and they matter for anyone hoping a stronger rewriter will settle the matter. Held-out fit does not improve monotonically as rewriting models become more capable. Increasing compute at test time gives inconsistent gains. The privacy measured is empirical, probed through canaries, and the records are themselves synthetic medical records. The authors frame privacy-aware rewriting as a promising direction, not a solved one.

The comparison puts three familiar options on one footing. Training directly buys capability with leakage; differential privacy buys safety with capability; rewriting the data first, with an eye on what makes a record unique, does better than either at holding both. The evaluation framework is meant to carry forward, so that future models and methods can be compared on the same terms.