Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning
Changbo Yan, Zhongbo Zhang, Zaibin Zhang, Yifan Wang, Lijun Wang, Huchuan Lu
Dalian University of Technology
Published September 10, 2026 · on the feed September 15, 2026 · ▲ 7 · arXiv 2609.13318
The abstract
3D point-cloud observations are inherently ambiguous in complex, cluttered manipulation scenes, where target objects may be partially occluded or tightly intermingled with visually similar distractors. As a result, standard 3D diffusion policies often struggle to localize and exploit task-relevant geometry as scene complexity grows. We propose Attention-DP3, a spatially object-aware 3D diffusion policy that injects object-level geometric cues via attention while keeping the DP3 diffusion backbone unchanged. Our pipeline performs open-vocabulary 2D segmentation on RGB images, then lifts predicted target masks into 3D using calibrated camera geometry to obtain object-centric geometric priors. We incorporate these cues through Tri-field Attentional Conditioning, which constructs three complementary fields: (i) a targetness field to anchor the target object, (ii) an intra-target saliency field to emphasize task-relevant geometry within the target, and (iii) a backgroundness field to suppress distractors and clutter. Experiments on Adroit, DexArt, MetaWorld, and the real-world SO101 platform show consistent improvements over DP3, achieving state-of-the-art performance across benchmarks. Notably, as distractor objects increase, DP3 drops sharply, whereas Attention-DP3 remains stable and outperforms DP3 by up to 31\% under heavy clutter. The code is publicly available at https://github.com/zhangzhongbo2213/Attention-DP3.
The Brief
2:32A short podcast summary of the paper. Written and voiced by AI.
Transcript
Attention-DP3. Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning. By Changbo Yan and five others, from Dalian University of Technology. Published on the tenth of September. Its code repository is worked out: about nine hundred and fifty source files, with a readme.
In complex, cluttered manipulation scenes, 3D point-cloud observations are inherently ambiguous. The target object may be partly occluded, or tightly intermingled with distractors that look much like it. As scene complexity grows, standard 3D diffusion policies often struggle to localize the geometry that matters for the task, and to make use of it. More clutter means more points that are not the target, and more chances for the policy to attend to the wrong object.
Attention-DP3 is a 3D diffusion policy made spatially aware of objects. It injects object-level geometric cues through attention, and it keeps the DP3 diffusion backbone unchanged. The policy itself is not redesigned. What it receives is extra information about where the relevant object sits in the scene.
The pipeline starts in two dimensions. Open-vocabulary segmentation runs on the RGB images, and the predicted target masks are lifted into 3D using calibrated camera geometry, which gives object-centric geometric priors. Those cues enter through Tri-field Attentional Conditioning, which constructs three complementary fields. A targetness field anchors the target object. An intra- target saliency field emphasizes the task-relevant geometry within the target. And a backgroundness field suppresses distractors and clutter. Segmentation in images is where the object is identified, and camera geometry is what carries that identification into the point cloud the policy actually works with. Read together, the three fields answer three questions: which object, which part of it, and what everything else is.
Experiments on Adroit, DexArt, MetaWorld and the real-world SO101 platform show consistent improvements over DP3, with state-of-the-art performance across the benchmarks. The sharpest contrast comes as distractor objects are added. DP3 drops sharply, while Attention-DP3 remains stable, and under heavy clutter it outperforms DP3 by up to thirty-one percent. The code is publicly available. That clutter result is the one the design was built to address, since the whole method is about finding the target among distractors.
Where the change lives is the telling design choice. The backbone that generates actions is left alone. What changes is what it attends to, with three fields telling it where the target is, which parts of the target matter, and what to ignore.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a robot that loses its target in a crowd.
You are training a robot arm that acts from 3D point clouds of the table in front of it. On a tidy table it does well.
The road ahead
- 1 · The problem ~6 min A robot that acts from 3D points works on a tidy table but loses track of its target once clutter and lookalike objects pile up.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Find the named object in the camera image, carry that mask onto each 3D point, and give the policy per-point fields as extra conditioning.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Segment the target in the image, lift the mask onto the points, build three fields, then feed them through attention into the unchanged DP3 denoiser.
- 6 · The experiment ~6 min You’ll guess: As more distractor objects were added to cluttered manipulation scenes, how did the standard DP3 policy and Attention-DP3 hold up?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.