Empirical Variational Autoencoder

Empirical Variational Autoencoder. By Kaede Shiohara, from the University of Tokyo. Published on the fifth of October. Its code repository is public and real: about twenty-five source files.

A conventional Variational Autoencoder constrains its latent space with a standard-Gaussian prior. For generating sequences, the abstract points to a cost of that choice: a gap between the latent distribution of the prior and that of the posterior, typically observed in conventional VAEs, which stands in the way of high-fidelity ancestral sampling.

Empirical Variational Autoencoder, or EVA, is a general generative framework for continuous-valued sequences, meaning ones that are not vector-quantized. It keeps the evidence lower bound of the VAE, but learns autoregressive latent priors empirically, from the training data. The fixed standard-Gaussian constraint gives way to priors the model predicts for itself. Rather than being told where its latents ought to sit, it learns where they actually do.

The implementation cost is small. According to the abstract, the whole change can be implemented by only an additional single linear layer on top of a VAE. With self-predicted priors in place of the standard Gaussian, EVA significantly alleviates the gap between prior and posterior, and that is what leads to high-fidelity ancestral sampling for sequential data generation.

Extensive experiments on image and sound synthesis show EVA achieving generation quality competitive with autoregressive diffusion baselines, despite much faster inference.

The abstract is short, and so is the claim it makes. A familiar model plus one linear layer, learning its own prior instead of assuming one, is put forward as matching autoregressive diffusion on images and sound while generating far more quickly. The idea fits in a sentence: let the prior be learned from the data.