Octrees as an Explicit 3D Language

Octrees as an Explicit 3D Language. By Ran Dan and five others, from Peking University and Independent Researcher. Published on the first of October. Its code repository is worked out: about three hundred source files, with a readme and a licence.

Existing 3D large language models make two compromises. The first is in what the model sees. Shapes are compressed into latent codebook indices or into coordinate text, and either way the spatial structure is removed from what the model observes. The second is in how the model learns. The 3D modality is acquired by fine-tuning the backbone, which overwrites the general language ability the backbone already had.

OctLLM answers both. On the input side, geometry enters as an explicit 3D sequence of octree occupancy tokens, so the structure of the shape is the thing being read. On the training side, 3D capacity is added in parameters kept separate from the pretrained ones, so the language pathway is left as it was.

Full octree sequences grow rapidly with depth, which makes them long. OctLLM shortens them by randomly emptying nodes at the penultimate level and omitting their descendants, in a way that preserves the shape. The result is a shorter Sparse Octree, or S-Octree, anchored by both coordinate and depth, and it serves position-aware mask-modeling generation as well as 3D understanding.

For the second front, the abstract sets out why the usual options fall short. Full fine-tuning is costly, LoRA limits 3D capacity, and both modify the language pathway. In OctLLM, mesh tokens are routed through independent trainable branches in a subset of blocks. Text and image tokens keep the frozen vision-language pathway. The two streams interact through shared self-attention, which is where the geometry and the language meet.

OctLLM trains far fewer parameters than full fine-tuning would. Even so, it sets a new state of the art among unified multimodal language models. Measured against ShapeLLM-Omni, it lowers image-to-3D FID by seventeen point four percent, and raises render-grounded captioning by twenty-eight point seven points. On general language benchmarks it matches the backbone it was built on.

The two halves of the design come from the same instinct: keep what is already structured intact. The shape keeps its spatial structure because the octree is the language. The backbone keeps its language ability because the new modality lives in branches beside it. The final result, matching the backbone on language while leading on 3D, is the evidence that the second instinct held.