Jag: Direct Box Prediction for Efficient Visual Grounding

Jag. Direct Box Prediction for Efficient Visual Grounding. By Xiuyuan Zhu and five others, from the University of Chinese Academy of Sciences, the State Key Laboratory of Communication Content Cognition, and one other institution. Published on the ninth of October. Its code repository is worked out: about fifty source files, with a readme and a licence.

Visual grounding means finding the part of an image that a phrase refers to. It combines understanding of image and language with a geometric prediction: the answer is a box. Autoregressive multimodal models often express that box as a sequence of coordinates, written out one token after another, the way they would write a sentence. That sequential decoding adds computation on top of the multimodal processing that localization already needs. The model has to understand the scene, and then spell out its answer piece by piece.

The authors see a different route. If the model's representations are adapted to box geometry, the box can be predicted directly. Jag does this. It adapts a compact pretrained multimodal model to predict a continuous box in one go, instead of generating coordinates as text.

A lightweight regression head reads the state of the last valid input token. That state is conditioned on the image and on the referring expression, so it already holds what the model knows about both. From that single state, the head predicts all four box coordinates together.

Training uses geometric supervision, and it adapts three parts jointly: the language backbone, the visual merger, and the regression head. Because of that, the localization losses shape two things at once, the multimodal representations and the mapping from those representations to coordinates. At inference, the image and language processing stays as it was. The box comes back in one forward pass, with no coordinate generation and no text parsing.

Across benchmarks, Jag consistently improves grounding accuracy over its base model, across the target sizes and expression groups observed. Component ablations show that adapting the representations and the head together substantially beats training the head alone, so the gain is not just the new head. Against existing specialized grounding models, Jag reaches competitive accuracy, with lower latency for a single request and lower GPU memory use.

The authors frame the result as supervision shaped to the geometric answer. A box is four numbers. Training the model toward those four numbers directly, rather than toward a string that spells them out, is what turns a compact multimodal model into one that grounds accurately and predicts efficiently.