F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

F2G-Pose. Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation. By Jinyu Zhang and four others, from Fudan University and the Shanghai Innovation Institute. Published on the twenty-second of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Category-level object pose estimation from RGB-D asks for an object's 3D rotation and translation without an instance-specific CAD model and without reference views. Only the category is known. That makes the task highly ambiguous: objects within a class vary, many are symmetric, and the camera sees only part of each one. Any one of those would leave room for more than one plausible answer. Together they make the answer hard to pin down from a single view.

Current solutions struggle to be both reliable and efficient. Methods that go through intermediate structures are prone to errors propagating from one stage to the next. Diffusion models suffer from high inference latency, and need complex candidate extraction after the fact. One family is quick but fragile, the other is more careful but slow, and neither gives both at once.

F2G-Pose is a real-time, direct framework that does the whole job in a single feed-forward pass. From one RGB-D observation it predicts the category-level pose, the object's metric size, and its completed shape in the camera frame, all at once. There is no intermediate structure for errors to travel through, and no set of candidates to sort through afterwards: one input, one pass, one answer.

It begins by lifting dense features from a visual foundation model onto partial point clouds. Local geometry is encoded with DGCNN tokenization, and a Mixture-of-Experts Transformer integrates structure at the level of the whole object. Predicting size jointly with camera-frame shape completion gives the network dense geometric supervision, which encourages metric consistency: the predictions are held to the object's real, measured scale.

On the training side, the split is clean. The RADIO backbone is frozen and was pretrained externally; the task-specific network is trained only on synthetic SOPE data annotated with pose. The part built for this task learns from synthetic data alone, and that is the backdrop for the results on real objects that follow.

With that training, F2G-Pose achieves state-of-the-art accuracy on SOPE and ROPE, and runs at twenty-eight frames per second for pose-only inference. It also shows strong camera-frame geometry on HouseCat6D, and transfer to real objects on HANDAL.

Further analyses cover latency, robustness, shape completion, and how GenPose++ behaves with its candidates, and the authors read them as confirming the method's efficiency and reliability. Real-world manipulation experiments, run with segmented RGB-D inputs, show its potential for deployment.

The argument is that one pass can be enough. Rather than chaining intermediate steps or sampling and sifting candidates, F2G-Pose leans on foundation features and dense geometric supervision to answer directly, fast enough for real time. The project page, code, and pretrained models are available.