LLM2R: From Large Language Models to 3D Reassemblers

LLM2R. From Large Language Models to 3D Reassemblers. By Zhouhan Zhong and four others, from the National University of Singapore and the Hong Kong Polytechnic University. Published on the thirteenth of October. No code repository is linked to it.

Three-dimensional part assembly starts from a pile of unassembled parts and tries to rebuild the complete shape, by predicting a rigid SE three transformation, a rotation and a translation, for each part. The catch is that a part's own geometry rarely settles where it goes. A leg could belong at any corner. So the task demands reasoning over the whole set of parts at once. Existing methods lean on pairwise relations between parts, or on iterative rounds of generation and verification, and both scale poorly as objects acquire many parts.

LLM2R recasts part assembly as sequence modeling, and hands that sequence to a pretrained, decoder-only large language model. Each part becomes a block of tokens. All the blocks are laid end to end. A model built to read long sequences then reads the entire assembly together, which is the global reasoning the task was asking for. There is no need to compare every part against every other part in turn, and no loop of proposing an assembly and checking it. The whole set goes in, and the poses come out.

Concretely, a Point-BERT encoder and a Q-Former compress each part into a block of geometric tokens. The blocks for every part are concatenated, and a causal LLM, adapted with Low-Rank Adaptation, or LoRA, processes the whole sequence in one forward pass.

Causal attention brings a problem of its own: a part's hidden states can only observe the parts that came before it in the sequence. The fix is a block of learnable summary tokens appended after the last part. Since those tokens follow every part, their states aggregate the whole assembly, and they do so at a constant token cost however many parts there are. A part-aware pose head then fuses each part's own states with these summary states before predicting that part's rotation and translation. The result is one architecture that handles both semantic and geometric assembly, and predicts every pose in a single pass that is not autoregressive.

On all three PartNet categories, LLM2R achieves the highest part accuracy and success rate. On PartNeXt it has the best rotation error, translation error and part accuracy. On Breaking Bad it has the lowest translation error and the fastest inference, and only a fracture-specialized method is more accurate there. Taken together, that is three different benchmarks, and on each one LLM2R leads on at least part of the scoreboard, with the single exception on Breaking Bad conceded to a method designed specifically for fractured objects.

The advantage widens as part counts rise. At one hundred parts, it exceeds GARF by thirty-three point five points of part accuracy on Breaking Bad, and exceeds Assembler by twenty-nine point one points on PartNet. It also transfers zero-shot to unseen categories, with part accuracy more than sixteen points above the strongest of three baselines.

The pattern in the numbers matches the diagnosis. Methods built on pairs of parts were said to struggle as part counts grow, and that is precisely where LLM2R's lead is largest. Treating an assembly as one sequence, and giving a causal model a place at the end where the whole of it can be seen, turns a combinatorial puzzle into the kind of long-context reading these models were built for.