When Left Is Right? Camera-Motion Understanding beyond Screen

When Left Is Right? Camera-Motion Understanding beyond Screen. By Chong Cheng and five others, from the Hong Kong University of Science and Technology, Guangzhou, Tencent, and one other institution. Published on the twelfth of October. Its code repository is public but slight: one source file.

Can a vision-language model tell left from right? Modern models of this kind solve visual puzzles and follow hour-long videos, yet asked whether the camera moved left or right, most do no better than a coin flip. Two fixes have been tried, separately: adding camera-motion data, and adding geometric representations. Neither closes the gap. The authors argue that camera-motion understanding needs two conditions at the same time. The representation must couple off-screen camera motion with on-screen flow dynamics. And it must be interpreted by the language model, not trained as a standalone pose or depth estimator.

CameraVLM is built to meet both conditions. It pairs a motion-state representation with state-language alignment. The first answers the coupling condition: it describes motion off screen and on screen together. The second answers the interpretation condition: the language model is the one that reads that state and gives it meaning.

The representation uses two kinds of token. Camera tokens capture how the camera moves off screen. Flow tokens capture how the subject and the background move on screen. A two-stage training strategy keeps language in the loop, so that both kinds of token are learned together with the semantics of camera motion, rather than as a separate estimator whose output the language model never shapes.

CameraVLM has only nine billion parameters. Across three benchmarks and more than twenty baselines, it outperforms all open-source and camera-motion-specific models by a wide margin. It also matches or surpasses GPT-6 Astra, and human viewers, in a study with forty-eight participants. The starting point was models doing no better than a coin flip on a question as simple as left or right. The endpoint is a model at that parameter count standing level with people on it.

The diagnosis does most of the work. Motion data alone and geometry alone had each fallen short, and the authors' reading is that both conditions have to hold together. CameraVLM is that combination: on-screen and off-screen motion in one representation, with language present while it is learned.