Geometric and Semantic Coupling for Interaction Understanding in 3D Scenes
Hanyang Kong, Xingyi Yang
National University of Singapore · The Hong Kong Polytechnic University
Published September 21, 2026 · on the feed September 23, 2026 · ▲ 10 · arXiv 2609.25247
The abstract
Interaction understanding in 3D scenes requires a joint description of movable parts, their motion, and the regions through which they can be operated. We present Segment-Snap, which connects these outputs through the physical relationship between parts and handles. Learned predictors identify broad part surfaces and small handles. A geometric decoder uses planar and upright priors to constrain motion, then selects hinge lines using predicted handle locations, without training a motion regressor. Conversely, a joint part-and-handle predictor supplies additional handle candidates, whose motion classes are refined using containing parts. Each information transfer is applied once, without iterative feedback. On Articulate3D validation, handle guidance raises motion-gated AP from 13.74% to 40.98% at fixed masks and axes. Additional handle candidates raise handle AP from 24.63% to 29.65%; part-based class correction adds 0.98 points, and full context reaches 30.99%. Repeated training, learned-decoder controls and paired visualizations establish the benefits and limitations of combining geometric and semantic evidence for interaction understanding.
The Brief
2:00A short podcast summary of the paper. Written and voiced by AI.
Transcript
Geometric and Semantic Coupling for Interaction Understanding in 3D Scenes. By Hanyang Kong and one other, from the National University of Singapore and the Hong Kong Polytechnic University. Published on the twenty-first of September. Its code repository is worked out: about fifty source files, with a readme and a licence.
Understanding how to interact with a 3D scene needs a joint description of three things. Which parts can move. How they move. And the regions through which a person would operate them, such as a handle. Those outputs depend on one another, and describing them separately loses that connection.
Segment-Snap connects the three through the physical relationship between parts and handles. Learned predictors pick out both the broad surfaces of parts and the small handles. Information then flows in both directions between them.
In one direction, a geometric decoder uses planar and upright priors to constrain the motion, then chooses hinge lines using the predicted handle locations, with no motion regressor trained at all. In the other direction, a joint part-and-handle predictor supplies additional handle candidates, and the motion classes of those candidates are refined using the parts that contain them. Each of these transfers is applied once, with no iterative feedback.
On Articulate3D validation, with masks and axes held fixed, handle guidance raises motion-gated AP from thirteen point seven four percent to forty point nine eight percent. Adding the extra handle candidates raises handle AP from twenty-four point six three percent to twenty-nine point six five percent. Part-based class correction adds zero point nine eight points, and the full context reaches thirty point nine nine percent.
Repeated training, controls using a learned decoder, and paired visualizations establish both the benefits and the limitations of combining geometric and semantic evidence in this way.
The idea is that parts and handles explain each other. A handle says where a hinge is likely to be, and a part says what kind of motion a handle belongs to. The largest single gain comes from letting handles guide motion.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A still scan of a kitchen, and a question it cannot answer by shape alone: which way does that door swing?
You are given a scanned room as a coloured point cloud. Your job: say which cabinet doors and drawers can move, how they move, and where a person would grab them. Nothing moves during the scan.
The road ahead
- 1 · The problem ~7 min From one still scan of a room, you must say which parts move, how they move, and where to grab them, but shape alone leaves gaps.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Handles and parts are physically joined, so the handle picks a door's hinge side and the part tells the handle how it moves.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Predict parts and handles, narrow each part's motion with flat and upright assumptions, then let the nearest handle pick the hinge line.
- 6 · The experiment ~6 min You’ll guess: With part masks and axes held fixed, motion-gated AP on Articulate3D validation starts at 13.74%. Where does adding handle guidance take it?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.