UltraTex: Unleashing 2K Multi-View Diffusion for 3D Texturing

UltraTex. Unleashing 2K Multi-View Diffusion for 3D Texturing. By Yibo Zhang and six others, from Jilin University, the Shanghai Innovation Institute, and four other institutions. Published on the nineteenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Realistic, production-ready 3D assets need high-quality textures. Multi-view diffusion has shown promise for texturing an object from a reference image, but these methods usually operate at low resolutions, such as five hundred and twelve or seven hundred and sixty-eight. At that size, the high-frequency detail in a high-resolution reference image is hard to keep. The obvious fix, scaling up to a resolution of two thousand and forty-eight, is computationally prohibitive. The unified multi-view sequence then exceeds two hundred and twelve thousand tokens, with excessive memory use and latency to match.

UltraTex is an end-to-end framework that makes high-resolution multi-view texturing efficient. Its key observation is that renderings of a single object from several views are redundant in two particular ways. One is redundancy in the sequence caused by background. The other is that interactions between tokens in the foreground are sparse. Each form of redundancy gets its own remedy. Put together, the aim is to spend the model's compute on the object being textured rather than on the empty space around it, and within the object to exploit how sparse the token interactions already are, so that the jump to two thousand and forty-eight stops being prohibitive.

Background Token Dropping removes background tokens before they reach the DiT backbone. Block-Sparse Attention then reduces attention computation over the foreground sequence that remains. Running inference on the foreground alone raises a risk of reconstruction artifacts, so the authors add a third piece, Foreground-Aware VAE Decoding, to ensure the final high-resolution views keep their quality. The three pieces come in sequence: prune the sequence, thin the attention, then decode carefully.

Training diffusion at two K resolution across many views is also demanding on data. To meet that, the authors construct G-buffer TexVerse, a large-scale multi-view rendering dataset at ultra-high resolution, covering over two hundred and sixty-eight thousand 3D assets.

The reported textures are visually faithful and rich in fine-grained detail, and the efficiency gains are large. Compared with the baseline, on common samples from the authors' dataset, training speeds up by between twenty point six times and ninety-one point one times. End-to-end inference speeds up by between twenty-two point three times and seventy-four point six times.

The approach earns its speed by declining to compute what does not need computing. Background tokens are dropped outright, attention over the foreground is made block-sparse, and the decoder is adapted so that working on the foreground alone does not cost image quality. The dataset is the other half of the contribution, since a resolution this high is only reachable if there is data at that resolution to train on.