RealMe: Immersive Volumetric Video from Minimal Captures

RealMe. Immersive Volumetric Video from Minimal Captures. By Chenxu Zhou and six others, from Zhejiang University and Four D V dot A I. Published on the twenty-first of October. Its code repository is a placeholder: a readme, and nothing to run yet.

The goal is immersive volumetric video made from sparse, low-resolution captures. Conventional dense multi-view reconstruction can already produce high-quality volumetric video, but it relies on costly capture setups, and that makes it impractical in real-world scenarios. More recent approaches use generative models to relax those capture requirements. They still suffer from two problems, though: limited resolution, and inconsistency in 4D, meaning across viewpoints and over time at once. A video that looks sharp from one camera but disagrees with its neighbours, or flickers from frame to frame, cannot be reconstructed into a convincing volume.

RealMe is a unified framework built around a high-resolution video diffusion model. It takes limited captures and enhances them into 4D-consistent videos at two K resolution, which then feed volumetric video reconstruction. The heavy lifting moves from the capture rig to the model, which is asked to supply what the sparse captures leave out.

The pipeline runs in a few steps. RealMe first constructs auxiliary data from the capture videos, and encodes it into latents together with layout-aware positional embeddings. The video diffusion model then adapts factorized spatio-temporal attention to the multi-view layout of the capture. It denoises those latents across viewpoints and across time, and decodes them into enhanced videos that refine the 4D representation. Fitting the attention to the capture layout is what helps the model capture temporal dynamics and multi-view relationships, and that is where the gain in 4D consistency comes from.

One more piece addresses a different gap: the one between generated videos and real captures. To reduce it, a latent frequency loss supplies supervision in the frequency domain, applied directly in latent space.

Extensive experiments cover the public DNA-Rendering benchmark and the authors' own self-captured RealVV dataset. On them, RealMe creates immersive volumetric videos and outperforms state-of-the-art methods in both visual fidelity and 4D consistency.

The trade the paper proposes is plain. Rather than paying for dense capture hardware, capture little, and let a diffusion model that understands the camera layout supply resolution and consistency afterwards. Its results are reported on exactly the two qualities that earlier generative approaches lacked, resolution and 4D consistency. Each part of the design maps onto one of the original complaints. The high-resolution model answers limited resolution. Attention fitted to the capture layout answers inconsistency. And the latent frequency loss narrows the remaining distance between what is generated and what a camera would really record.