SAGE: Salient Factor Discovery and Generation with Visual Foundation Representations

SAGE. Salient Factor Discovery and Generation with Visual Foundation Representations. By Shuang Liang and seven others, from the University of Hong Kong, Boston College, and two other institutions. Published on the thirtieth of September. No code repository is linked to it.

Contrastive analysis starts from two datasets. A target set, say faces wearing eyeglasses, and a background set, faces without them. The task is to separate the salient factors specific to the target from the common content both sets share. The authors want more than a yes or no for the glasses. They want salient representations that capture the target-specific detail in each image: the shape, colour and position of the glasses. Representations like that could reveal subtypes without any subtype labels, and could guide the generation of new examples of a subtype that was discovered, even one that has no name or text description.

SAGE learns both factors, salient and common, directly in the high-dimensional spatial latent of a frozen representation autoencoder. It then conditions a diffusion transformer on the learned salient representation of a reference image, so that a discovered subtype can be generated by pointing at an example of it.

Two pieces carry this. The frozen representation autoencoder supplies the space in which salient and common factors are separated, and the diffusion transformer turns a salient representation back into new images. Discovery happens in the first, generation in the second.

On Digits-ImageNet and on FFHQ eyeglasses, SAGE combines high-fidelity reconstruction, with rFID below two, with unsupervised subtype discovery. It recovers the digits better than baselines, with probe accuracy of zero point nine five zero against at most zero point two eight one. On the faces, it reveals eyewear types, finer sunglasses styles, and mislabeled images.

Conditioning generation on the salient representation raises Digits-ImageNet subtype accuracy to ninety point five percent, against twenty-seven point seven percent with the unfactorized latent, and raises diversity on both datasets. On retinal OCT scans, SAGE's salient space separates three diseases using only labels of normal versus disease.

The retinal result shows what the setup buys. The only supervision was a coarse split between healthy and not, and the salient space still pulled apart three distinct diseases. Subtypes nobody labelled become visible, and the same representation that reveals them can be used to generate more of them.