Does Native 3D Texture Generation Necessarily Require 3D Assets for Training?

Does Native 3D Texture Generation Necessarily Require 3D Assets for Training?. By Jiangshan Wang and eight others, from MMLab at the Chinese University of Hong Kong, Tencent Hunyuan, and four other institutions. Published on the twenty-eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Native 3D texture generation paints colors directly in 3D space onto a given geometry, guided by reference images taken from several views. The general belief has been that training such a model needs real 3D asset data, in large quantity and of high quality. Acquiring that data is described as a long-standing and challenging problem, one that has not gone away. If the belief is right, then texture generation can only ever scale as far as the supply of 3D assets lets it, and that supply is the hard part.

Tex-Zero challenges the belief directly, by training a high-fidelity native 3D texture generator with no 3D assets at all. The key observation is a split between two kinds of information a training sample carries. What texture training really needs is high-quality, fine-grained color. The geometric information is less critical, and it can be constructed by hand rather than taken from real assets. If that holds, the problem changes character. Two dimensional images are abundant and of high quality, and they could be turned into effective training samples for 3D texture generation.

The conversion works like this. Each 2D image is represented as a plane placed in 3D space. Patch-wise random rotations and aggregation then assemble those planes into complex geometric structures, so that a flat picture becomes a training sample with shape to it. The color is real; the geometry is manufactured. On this constructed data the authors train the Tex-Zero VAE. It can reconstruct real 3D assets with high quality, despite never having observed one during training.

On top of the VAE they train the Tex-Zero DiT, again exclusively on the constructed image data. The conditioning multi-view images get the same treatment as the training data: they are turned into planes in 3D space and encoded by the same Tex-Zero VAE. Passing the condition and the target through one shared encoder reduces the representation gap between them, and the authors report that this improves generation quality.

Extensive experiments show Tex-Zero producing high-fidelity 3D textures with fine-grained details, using images alone as training data. The abstract gives no figures and names no baselines, so the evidence it states is qualitative: high-fidelity output from a pipeline that never saw a real asset in training, and a VAE that reconstructs real assets well all the same.

The title asks a question, and the answer offered is no. Native 3D texture generation does not necessarily require 3D assets for training, because the part that matters most, rich color, can come from ordinary images, and the part that matters less, geometry, can be constructed by hand. The authors frame this as a promising perspective on the data paradigm for scaling 3D texture generation, one in which the scarce resource is no longer the one the field assumed it needed.