GRACE: Generation-aware latent compression for efficient video generation

GRACE. Generation-aware latent compression for efficient video generation. By Jiyoung Kim and seven others, from KAIST AI and Kakao. Published on the seventh of October.

One effective way to speed up a video diffusion model is to compress its latent harder. With a highly compressed video autoencoder, the Diffusion Transformer, or DiT, operates on far fewer tokens. The trouble is that such autoencoders are hard to train. A higher compression ratio degrades reconstruction quality, and recovering that quality requires more channels, which is known to slow the convergence of the DiT.

There is a second cost. The compressed latent is not the one the DiT was trained on, so a pretrained DiT has to be either retrained from scratch or adapted at considerable expense. An apparent way out is to compress the very autoencoder the DiT was trained with, which looks as though it should preserve compatibility. But if that compression is optimized for reconstruction alone, the latent still drifts away from the distribution the DiT has learned.

Generation-Aware Latent Compression for Efficient Video Generation, GRACE for short, is a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. The central move is in the name. The autoencoder is optimized for generation, not for reconstruction alone.

In the first stage, a frozen base latent is kept from the pretrained encoder, and a residual latent is learned on top of it to carry the information lost under stronger compression. At the same time, the compressed latent is aligned with the pretrained latent in the feature space of the frozen DiT. Alignment is measured where the generator sees it, which is what makes the autoencoder generation-aware.

In the second stage, the DiT itself is adapted, but only with lightweight fine-tuning, and with what the authors call asymmetric denoising. The base latent, the one inherited from the pretrained encoder, is denoised ahead of the residual that was learned for the extra compression.

The test case is Wan2.1-I2V-14B. GRACE cuts its token count by eight times, and its latency by eleven point one times, at a size of four hundred and eighty by eight hundred and thirty-two by eighty-one. At that speed-up, it matches the generation quality of the original pipeline before compression, as measured on VBench.

The framing is about where compatibility is lost. Retraining the DiT is expensive, and compressing the original autoencoder for reconstruction alone quietly moves the latent anyway. GRACE keeps the original latent as a frozen base, adds only what compression removed, and checks alignment through the eyes of the frozen DiT, so the generator is asked to adapt as little as possible.