USDCraft: Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation

USDCraft. Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation. By Chuanrui Zhang and six others, from NVIDIA, NTU, and one other institution. Published on the eighth of October.

Real-to-sim robot manipulation depends on articulated 3D assets that are both geometrically faithful and functional, because a policy trained in simulation has to transfer to the physical object. Recent mesh-based methods learn to infer articulation from annotated 3D assets. Deployment is where they struggle. Real objects can fall outside the training distribution, and their meshes can be incomplete or corrupted. A method that learned its answers from clean, annotated examples has little to go on when the object in front of it is unfamiliar, or when half of its surface was never captured. Those are exactly the conditions a real robot meets, which is why the gap between benchmark and deployment matters here.

USDCraft reformulates the task. Instead of predicting articulation from a mesh, it treats articulated asset reconstruction as programmatic modeling, grounded in partial geometric evidence. A pretrained language model writes and revises executable programs that describe simulation-ready articulated assets, and it does this without any task-specific training.

Two proposed components let the language model reason about geometry. The first is source geometry analysis. It converts the source mesh into a metric textual description, and that description distinguishes surface that was observed from space that is unknown. The second is iterative geometric rechecking. Each candidate program is re-encoded into the same representation, so any discrepancy against the source points to an edit in the program, while regions that were never observed stay open to completion.

Visual feedback and physical authoring guidance complete the modeling loop. What comes out is an articulated USD asset with explicit physical properties, one that loads into Isaac Sim without manual adjustment. That last point is the practical test of the whole pipeline. An asset that needs a person to repair it before simulation would undo much of the benefit of generating it automatically.

Experiments report leading articulation recovery on two benchmarks. The authors also validate USDCraft in real-to-sim-to-real robot manipulation, carrying the reconstructed assets from a real object into simulation and back to a physical robot.

The move here is from learned inference to written programs. A model trained on annotated assets is bounded by what it saw. A language model writing code against a textual account of the geometry, and checking each draft against that same account, can say exactly where the program disagrees with the evidence, and can leave the unseen parts honestly open.