Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

Super-Resolution in The Right Latent Space. A Frozen Vision-Foundation Substrate. By Wanzhou Lei and eight others, from the University of California, Berkeley, Alibaba Group, and three other institutions. Published on the eighth of October. No code repository is linked to it.

Start with a picture of the problem. In an image latent space, the embeddings of sharp, natural, high-resolution images form a manifold: a surface where all the good images live. Degrading a high-resolution image pushes its embedding off that surface. Real-world super-resolution then becomes a mapping task. Take the degraded embedding and bring it back onto the manifold. Not to any point on it, though. It has to land on the one point that preserves what the input still carries, both its semantics and its pixel details. Landing anywhere else gives a sharp image of something that was not there.

Every published method, the authors say, performs this mapping in one of two places: in pixel space, or in a latent space built for reconstruction. Their claim is that both are the wrong substrate for the job. In such spaces, low-resolution and degraded images are embedded far from the manifold, so the mapping is difficult and expensive. And because these substrates carry little semantic information, there is not much to steer by when looking for the faithful point. When the degradation is heavy, the model goes astray, and the result is severe hallucination. The conclusion they draw is that the space you restore in is crucial to the task, not a detail.

The proposal, then, is to change the space rather than the mapper. The authors show that one suitable space is the latent space formed by fusing twenty-three layers of a frozen DINOv3-L. Two properties make it fit. First, degraded images are embedded near the manifold to begin with, so there is less distance to travel. Second, the substrate holds a hierarchy of information, running from a record of the pixels at one end up to semantics that are robust to degradation at the other. That hierarchy is what guides the model toward the faithful point, instead of a plausible but invented one.

The vision-foundation model itself stays frozen. On top of that frozen substrate, a decoder of four hundred and fifteen million parameters is trained, under two objectives: reconstruction and adversarial. Its job is to map the degraded embeddings back toward the manifold and decode the result to pixel space, and it does both in a single pass. The resulting model is called RAESR.

RAESR is compared against state-of-the-art restorers of two kinds, adversarial and diffusion-based. The test sets are RealSR, DRealSR, LSDIR and DIV2K-Val. Across them, RAESR attains the best trade-off between fidelity and perception, meaning it stays faithful to the input without giving up how real the output looks. It runs at thirty-seven milliseconds per image of five hundred and twelve by five hundred and twelve pixels, on a single H20 GPU.

The cleanest test of the central claim is a swap. Keep the recipe exactly the same, but replace the frozen substrate with a VAE latent. That version loses on every metric. Since only the substrate changed, the difference can be laid at the substrate's door.

The argument runs from geometry to engineering. If restoring an image means returning to a manifold, then the space you restore in decides two things: how far you must travel, and how well you can tell which point is the right one. Here, a frozen vision-foundation space shortens the trip and supplies the semantics to navigate by. A single-pass decoder does the rest.