Beyond the Current Scene: Event-Referential Grasping with Active View Selection

Beyond the Current Scene. Event-Referential Grasping with Active View Selection. By Hyunjoon Lee and six others, from Seoul National University and NVIDIA. Published on the thirtieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Imagine a robot that has watched people handle objects, and is later asked to pick something up. The request may not name the object or describe how it looks. It may refer to the role the object played in an earlier event, the one somebody used a moment ago, say. And by the time the request arrives, the object may no longer be visible. Grasping from the current scene alone cannot satisfy that request, because both the meaning of the request and the location of the target lie partly in the past.

BeyondSCe is a zero-shot robotic grasping system built for this event-referential setting. It takes two inputs, the event history and the current scene, and from them identifies the requested object, or the requested part of one, and localizes it for grasping. The history is what lets it resolve a request phrased in terms of a past event, and the current scene is what tells it where to reach now. It relies on pretrained models and uses no additional task-specific training, which is what zero-shot means here: nothing in the system was trained for this particular kind of request.

Occlusion is handled actively. When the target cannot be seen, the system recovers an event prior from the history and combines it with the geometry of the current scene. Together these guide the choice of camera viewpoints that are likely to reveal the target, so the robot looks where the past suggests the object went, rather than searching blindly. Two sources of evidence are joined here. The history says roughly where the target ought to be, and the geometry in front of the camera says which viewpoints could actually see that place.

Real-robot experiments use a single wrist-mounted RGB-D camera. For targets that are initially visible, grasp success is seventy-six percent, against forty percent for the strongest baseline. For occluded targets it is seventy-seven percent, against fifty-five percent for the strongest baseline in that condition.

Four additional scenes with heavy occlusion compare it to an active-perception baseline that is given the target's ground-truth 3D bounding box. Grasp success rises from seventy-five percent to ninety-five percent, and the mean number of views falls from three point three five to two point two.

A last detail sharpens the result. The baseline in that heavy-occlusion test was told exactly where the target was, and BeyondSCe still grasped more often with fewer looks. Knowing what happened earlier, it seems, can stand in for being told where to look.