Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding

Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding. By Xin Zhang and six others, from the National University of Singapore, the University of Chinese Academy of Sciences, and one other institution. Published on the twenty-first of October. Its code repository is worked out: about six hundred source files, with a readme and a licence.

Fine-grained multimodal intelligence runs in two directions. A model should be able to understand a designated region, and it should be able to localize a region from a piece of language. Existing methods learn the two directions separately, each with its own objective and its own task-specific annotations. Yet the two are inverses of one another. Describing a region and finding the region a description refers to are the same relationship read in opposite directions, and learning them apart leaves that relationship unused.

Generalized CycleGRPO couples them in one reinforcement learning framework, which the authors describe as the first to do so across spatial, temporal, and spatio-temporal domains. The coupling is a cycle. Given a region, a multimodal large language model generates candidate descriptions, then localizes each description, trying to reconstruct the region it started from. How much the reconstruction overlaps the input becomes a verifiable reward for the captioning direction. Rewards on each individual localization improve the inverse direction. A description that leads back to the right region has, in effect, checked itself.

What lets one cycle cover images and video is a common formulation of region space. The same loop extends from image masks and boxes to video intervals and to timestamped spatial evidence. During reinforcement learning, it needs neither reference captions nor an external language-model judge. In video the cycle has a particular effect: it encourages descriptions that tell a target event apart from similar recurrences of it. To measure that temporal distinctiveness, the authors introduce DOVE, short for Distinctive Occurrence-level Video-caption Evaluation, a human-verified contrastive benchmark.

On images, CycleGRPO raises SAMTok's overall gIoU on GroundingSuite from fifty-seven point five percent to sixty-eight point two percent. On GRES, it lifts average gIoU from seventy-one point two percent to seventy-eight point seven percent.

On temporal tasks, the comparison is with the Qwen3-VL model the training starts from. CycleGRPO reaches fifty-one point nine percent tIoU on Charades, against forty-five point six percent for the initialization. On DOVE it reaches thirty-three point three percent Temporal Evidence Coverage, against fourteen point eight percent.

The authors take these results to establish region and text cycle consistency as a general principle of self-evaluation for multimodal understanding and localization. Researchers who want region understanding and localization to improve together, without reference captions or a judge model, are the natural audience, and the models, the code and DOVE are all released on the project page.

Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding. By Xin Zhang and six others, from the National University of Singapore, the University of Chinese Academy of Sciences, and one other institution. Published on the twenty-first of October. This is the long version: the whole paper rather than the abstract.

Many questions about images and videos concern one object, one moment, one tracked entity. A model must describe a designated region well, and must find a region from a description. Existing reinforcement learning trains these two directions separately. Describing is trained against expensive written captions, or scored by an extra critic model. Finding is trained against geometric targets. The authors call this an avoidable supervision bottleneck, because the two signals describe the same correspondence and yet never constrain one another. A fluent caption need not identify its source region, and a strong localizer gives no signal back to the captions it reads. In video the problem sharpens. An event can recur several times, so a generic but correct caption can match three or four intervals equally well.

The starting point is the authors' own earlier conference work, which introduced CycleGRPO for image regions. Begin with a mask or a box. The model writes a caption, then uses that caption to find the region again. How much the found region overlaps the original becomes a reward that can be checked by geometry alone. No reference caption is needed and no judge model reads the text. The new question is whether that cycle extends beyond flat images. The answer is abstraction. A region becomes any element of a region space, provided the space supplies two things: a way to write regions as tokens the model can read and emit, and a bounded score from zero to one saying how well a predicted region recovers the target. Image masks and boxes, video intervals, and timestamped boxes are then three instances of one recipe.

The rollouts are nested. For one target region, the policy samples a group of candidate captions. Each caption then conditions its own group of localization attempts, and each attempt is scored against the target. The caption's reward is the average of those scores, because a single decoding might land on one of several plausible modes, and averaging estimates how reliably the caption identifies its source. In the temporal and mixed video settings, four captions each get four localization attempts; the spatial setting used six of each.

The same policy plays both roles, actor and then critic, and both roles are trained. The advantages are normalized at two levels. Captions compete only against other captions for the same target. Localization attempts compete only against siblings that read the same parent caption, so they are judged on accuracy under a fixed description. Both losses share parameters and are accumulated into one update with equal weights.

Two details guard against cheating. Captions that leak coordinates or timestamps are penalized, and timestamps are stripped before the localization prompt is built. For timestamped boxes, predicted and target anchors are matched one to one, and a pair counts only if their times fall within three seconds. A similar object at the wrong moment contributes nothing.

To measure distinctiveness, the authors built a benchmark called DOVE. Each group is one video with two to five occurrences of the same query, one hundred and eighty groups in all, with five hundred and thirty target intervals. Gemini 3 Pro proposed facts, humans corrected them over several rounds, and each target got private multiple-choice probes about order, state and outcome, frozen before any model's captions were inspected. A fixed text-only judge, which never sees the video, reads a caption and decides whether it supports the right answer.

The headline gains are real but uneven. On images, starting from SAMTok, overall gIoU on GroundingSuite rises from fifty-seven point five to sixty-eight point two percent. On DOVE, the score rises from fourteen point eight to thirty-three point three percent over the Qwen3-VL starting point, with a bootstrap interval resampled over whole video groups.

The comparison that most directly tests the thesis is the objective ablation. Captioning-only training, scored against reference captions, lifts DOVE only to seventeen percent. Localization-only training reaches twenty-nine point four. The full cycle reaches thirty-three point three, and also has the best Charades temporal IoU of the three. A second ablation keeps the cycle rollout but drops one reward side. Each side alone lands near thirty percent; only together do they pass thirty-three. On the spatio-temporal Open-o3-Video setting, the single-direction variants improve TVGBench but are less consistent across benchmarks, while the full cycle is best on all five.

The paper has a limitations section, and it is specific. Being reconstructable does not make a caption complete, fluent, or free of unsupported detail. The anti-leakage rules block obvious shortcuts but cannot enumerate every hidden one. The loop assumes the starting policy can already do both mappings; when localization is unreliable, the caption side gets a noisy signal, which the authors say helps explain why gains vary by initialization. From Qwen3-VL, QVHighlights is essentially flat, and on GAR-Bench position accuracy falls. The nested rollouts cost more than one-directional training. DOVE covers only visually distinguishable events inside given intervals, with no audio and no boundary discovery. Two further points come from the paper's own figures: the appendix shows failures such as a sea otter chasing turkeys described as a man chasing a deer, and apart from the DOVE bootstrap, the tables report no error bars or repeated training runs.

The idea to carry away is that describing and finding are one correspondence seen from two sides, so each can grade the other. A caption is good if it leads back to where it came from, and a localizer is trained on the very captions it is grading. Once regions are defined by a serializer and an overlap score, that loop works for masks, intervals and timestamped boxes alike, using region annotations alone. In video, its sharpest effect is on distinctiveness: saying not just what happened, but which time it happened.