ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

ReGain. Restoring Subject Fidelity in Personalization on Synthetic Images. By Shubhang Bhatnagar and three others, from the University of Illinois Urbana-Champaign. Published on the thirtieth of September. No code repository is linked to it.

Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of images of it. Increasingly, those images come from a diffusion model rather than a camera. The claim is that this matters. Fine-tuning on synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail. And the degradation is traced to one specific place: classifier-free guidance. Those two symptoms are visible to anyone looking at the output. The contribution is to follow them back to a mechanism rather than stop at describing them.

To isolate the cause, two models are fine-tuned from the same base model with the same DreamBooth recipe. One is trained on real photos of a subject. The other is trained on synthetic images of that subject, generated by the first. Then their guidance is compared. For the model personalized on synthetic images, the angle between the conditional and unconditional noise predictions is much larger than for the model personalized on real photos, and with it the norm of their difference. That inflation grows toward high frequencies. It also appears at other prompts semantically close to the subject, such as its class noun, but not at unrelated ones.

The diagnosis leads directly to a fix. ReGain is a training-free correction applied at sampling time. It measures how much each frequency band of the guidance is inflated relative to the base model, and scales that band down accordingly. It needs no real photos, which suits the setting, since the subject was personalized from synthetic images in the first place. The base model supplies the reference instead. On Stable Diffusion v1.5, ReGain closes fifty-one to sixty-four percent of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I.

The edges of the claim are drawn by the experiments. The controlled comparison holds the base model and the recipe fixed, so the only difference between the two personalized models is where their images came from. The inflation is not spread across every prompt either. It appears near the subject and its class noun, and not at unrelated prompts. The figure for how much of the gap is closed is reported on Stable Diffusion v1.5. On SDXL and SD 3.5, ReGain also improves subject fidelity, and text alignment is preserved on all three backbones.

As synthetic images become the material personalization is built from, the cost shows up somewhere specific. Not in the look of the training images, but in the guidance signal at sampling time, inflated most at high frequencies. That is also where ReGain corrects it, one frequency band at a time, and without a single real photo of the subject.