SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling

SeMo3D. Unifying 3D Generation, Editing, and Completion via Selective Modeling. By Xiaokun Sun and two others, from Nanjing University. Published on the second of November.

Three jobs in 3D content creation are usually handled as separate problems. Generation makes an asset from nothing. Editing changes part of an existing asset. Completion fills in a part that is missing. Yet all three are typically treated the same heavy way: regenerate the complete 3D asset under task-specific constraints, even when only a local region actually needs to change. That habit has three costs. The tasks stay isolated from each other. Content that should have remained untouched drifts and becomes inconsistent. And each task comes to depend on its own paired training data, before and after examples of edits or completions, which have to be found or built specifically for it.

The authors start from a single observation. All three tasks synthesize a designated region while keeping everything else intact. For generation that region is the whole asset. For editing it is the target region. For completion it is the missing region. They call this paradigm Selective Modeling, and they build SeMo3D on top of it. Seen this way, the three tasks are one task with a different region selected, so unification comes naturally. Unchanged content is preserved by construction, because it is never the thing being synthesized. And training no longer needs paired editing or completion data, since any ordinary 3D asset can be split into a selected region and the context around it.

Two pieces make this work. The first is a training procedure called region-selective denoising. On standard 3D assets, it randomly selects regions of varying sizes and applies noise and loss only to those regions. Because the selected region can be anything from a small patch to the full asset, a single training run learns all three tasks jointly.

The second piece is the architecture, named Omni3DiT. It is a multi-condition 3D diffusion transformer. It distinguishes the regions it has to create from the regions it has to retain, and it fuses three kinds of guidance: image, text, and point guidance. That combination is what the authors credit for fine-grained control over what gets synthesized.

The method is tested across three benchmarks, and the abstract reports that SeMo3D surpasses state-of-the-art methods on all of the tasks. In generation, the gain described is multimodal controllability. In editing and completion, it is strong consistency and accuracy, which is precisely where the regenerate-everything habit was said to fail. Beyond the benchmarks, the authors show SeMo3D working as a unified modeling tool for agents, so that 3D creation can proceed efficiently through natural-language interaction.

The move here is a reframing more than a new component. Once generation, editing and completion are described as choosing which region to synthesize, the separate pipelines, the drift in untouched content and the hunger for paired data all trace back to a single assumption, that every task has to rebuild the whole asset. Drop that assumption and one model, trained on plain 3D assets, covers all three.