EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

EyeRobot 2.0. Active Gaze for Precise Manipulation without Wrist Cameras. By Kush Hari and nine others, from UC Berkeley, Amazon FAR, and one other institution. Published on the second of October. No code repository is linked to it.

Standard policies for fine-grained manipulation with two arms are trained with an egocentric camera plus cameras on the wrists. Remove the wrist cameras and those policies pay for it. With passive stereo alone, real-world success drops from fifty-two percent to twenty-seven percent. The question is whether a single stereo camera can do the job if it is used the way a person uses their eyes.

EyeRobot two point zero is a framework, inspired by human vision, that uses active gaze to make fine-grained bimanual manipulation possible with only one stereo camera. It physically attends to a three-D fixation point in the scene, swivelling two eye viewpoints so that both centre their gaze on it. The resulting images are processed foveally: more visual tokens are allocated to the centre of each image, which concentrates computation on the features that matter for the task.

Active Visual Fixation, as they call it, only works if gaze is coordinated carefully while the task runs, and that coordination is learned in a hierarchy. First, a low-level gaze servoing policy is trained, conditioned on a goal object. Then a target selector is trained that emits fixation goals based on how far the task has progressed. Both are trained with reinforcement learning on real-world data. The servoing policy gets a dense geometric reward. The selector is co-trained with the behaviour cloning gripper policy, which lets it discover fixation sequences that can resemble the ones a human makes while doing the same task.

Fixation pays off a second time on the action side. Gripper information is canonicalised into an SE three frame defined relative to the fixation point. Because the eyes are already centred on what matters, describing the gripper from that point of view compacts the action distribution the policy has to learn. The same act of looking that shapes the visual input also simplifies the output. Gaze, in other words, is doing three jobs at once: choosing where to look, deciding where computation goes, and setting the frame in which actions are expressed.

The authors collected teleoperation data for seven real-world and six simulated tasks, and ran over one thousand physical robot trials and one thousand eight hundred simulated ones. EyeRobot two point zero was compared against passive stereo policies and against ego plus wrist camera policies, all trained on the same data. Holding the data fixed means the differences come from how each policy sees, not from what it was shown.

Using only stereo, EyeRobot closes the gap, outperforming passive stereo by forty percent in the real world and by twenty percent in simulation. Against ego plus wrist policies with clear wrist views, it matches them: sixty-nine percent against sixty-four. When a grasped object blocks the wrist cameras, it more than doubles their success, forty-eight percent against twenty-two.

The occluded case carries the argument. Wrist cameras help only while the object in hand leaves them a clear view, and when it does not, the single stereo pair with active gaze comes out well ahead. Two eyes that can point, with computation concentrated where they point, stand in for the extra cameras across these tasks.