LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

LeRF. Learning Reference Coordinate Frames for Perspective Taking Reasoning. By Bang Xiao and seven others, from the University of Illinois Urbana-Champaign, Shanghai Jiao Tong University, and two other institutions. Published on the twenty-eighth of September. Its code repository is worked out: about eight hundred and fifty source files, with a readme.

Perspective taking is a basic part of spatial intelligence. It means interpreting spatial relations from a specified viewpoint, such as that of another entity in the scene, or of an imagined observer. Vision-language models have grown more capable at spatial reasoning, but they still struggle here. When a question asks them to reason from a different perspective, they often fall back on the camera's viewpoint, answering as the photograph sees the scene rather than as the question asked.

LeRF trains vision-language models to build explicit reference frames and to use them when a question depends on viewpoint. Rather than asking the model to rotate the scene in its head, it has the model draw the frame and then reason over the drawing.

Given an image and a query, LeRF first decides whether a coordinate frame is needed at all. If it is, the model grounds the reference entity, and predicts the frame's origin along with a reference frame centered on that entity. A lightweight renderer then overlays the frame onto the image. Later reasoning works from these visual cues, with no external perception models and no explicit 3D reconstruction. The frame becomes part of what the model looks at. A question about how things are arranged from someone else's point of view is turned into a question about an image that now carries that point of view drawn on it.

Learning this happens in two steps. Supervised fine-tuning first teaches selective tool invocation, meaning when to call for a frame, together with predicting the frame itself. Reinforcement learning on spatial question and answer pairs then improves the reasoning that the frame guides. The first stage teaches the mechanics of making a frame. The second teaches the model to reason well once it has one.

Across diverse perspective-taking benchmarks, LeRF consistently improves over its backbone and performs strongly against existing open-source methods. Further evaluations show better grounding of the reference frame and better orientation estimation, which supports the case that the learned frames are doing the work in viewpoint-dependent reasoning.

The fix is to make the viewpoint visible. A model that kept defaulting to the camera is handed a drawn set of axes belonging to someone else, and reasons from there. It also learns when not to bother, since a frame is only drawn when the question needs one.