VLX-Go: Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinement

VLX-Go. Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinement. By Kyusong Lee and two others, from Om AI Research and the Binjiang Institute of Zhejiang University. Published on the fourteenth of October. Its code repository is worked out: about one hundred and eighty source files, with a readme and a licence.

Embodied visual tracking asks a robot to do two things together: pick out a designated target from what it sees, and steer its own motion to follow that target through an environment. Following one isolated person is a comparatively simple demonstration. Following the intended person when other people are around, the distractors, is still hard. Existing methods lose a substantial amount of success when they go from single-target tracking to distractor tracking. That drop matters most in the shared spaces that person-following robots have to move through, and that is where this work focuses.

VLX-Go is a vision-language waypoint policy built on one hypothesis: visual and language representations should be aligned before any tracking-specific motion is learned. Alignment first, behaviour second. The policy is meant to settle what it is looking at, and how that connects to language, before it is asked to learn how to move in response. Waypoints are its output: the policy predicts, from the robot's own point of view, where it should go next.

The policy combines complementary frozen visual encoders, multiscale temporal context, a transformer backbone, and a lightweight head that predicts waypoints relative to the robot. Training proceeds in three stages. First, vision-language alignment. Then multimodal reasoning mid-training. Finally, specialization for tracking, on nineteen thousand five hundred thirty-two oracle episodes. The order of those stages is the hypothesis put into practice: the general alignment of seeing and language comes first, and the narrow skill of following comes last.

A further step is explored after that. In on-policy refinement, the policy visits states of its own, and those states receive filtered oracle labels to learn from. Training then covers situations the policy itself creates, not only the ones the oracle episodes happened to contain. The authors describe this part as a preliminary study, and it should be heard with that label.

The baseline for comparison is the authors' earlier model, OmTrackVLA, which reaches forty-one point five four percent success on distractor tracking. The complete multi-stage recipe reaches fifty-five point eight percent.

On-policy refinement adds two point four two percentage points, reaching fifty-eight point two two percent, and it lowers the collision rate by one point nine nine points. Among the methods compared, the final policy records the highest success rates on both distractor tracking and ambiguous tracking.

Two claims come out of this, held with different confidence. The staged recipe, aligning vision and language before learning to move, is presented as a promising direction for tracking among distractors. On-policy refinement is offered more tentatively, as a result that motivates further study of person following in shared spaces.