InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators

InstantFusion. A Shared Latent Interface for Heterogeneous Image Generators. By Xuanlang Dai and seven others, from Fudan University, the Shanghai AI Laboratory, and two other institutions. Published on the ninth of October. Its code repository is public and real: thirteen source files.

Diffusion and flow models have become the standard approach to image generation, and they have produced a varied family of generators whose architectures and representations differ more and more. Each one generates well on its own. Much less studied is how their latent representations line up with one another, and whether their underlying denoising dynamics are consistent across models. In practice, each generator is its own closed system. Its intermediate states mean something only to itself, and a denoising run that starts in one model ends in that same model.

The starting point is an observation about compatibility. Generators trained independently show similar spatial organization in their latents. And handing an intermediate state directly from one model to another, with no alignment at all, can still preserve the coarse visual content. If unaligned models already agree that much, a light interface might make them agree enough to cooperate. InstantFusion is that interface: a lightweight shared latent space that connects the denoising trajectories of heterogeneous image generators.

Each generator gets its own Latent AutoEncoder. These project intermediate states from their generator into the shared space, and decode from the shared space into the latent space of a target generator. Two complementary objectives constrain the shared space. One aligns intermediate states that come from the same image at matched noise levels. The other synchronizes their corresponding local denoising velocities, so that states agree not only on where they are but on where they are heading.

Once trained, InstantFusion lets denoising pass seamlessly from one frozen generator to another. Because every model maps through the same shared space, there is no need for a dedicated mapping between every pair of models. Adding a generator means training its own encoder and decoder, not building a bridge to each of the others.

Its versatility is shown in four settings: acceleration, composition of multiple preferences, on-policy distillation across architectures, and composition across models. Performance is described as competitive across all of them. The range is the point of the demonstration: one trained interface serving tasks as different as speeding up generation and transferring behaviour from one architecture to another.

Where most work treats each generator as a closed system, InstantFusion treats them as stages that can be swapped mid-trajectory. The generators stay frozen. What is learned is only the common ground between them, and the observation that they already share some of it is what makes that ground small enough to learn.