PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

PANORAMA. Panoptic Grounded Captioning via Mask Proposal Selection. By Sara Pieri and four others, from Inria, the École normale supérieure, and three other institutions. Published on the sixteenth of September. Its code repository is worked out: about eighty-five source files, with a readme and a licence.

Systems that act in the world need image understanding that is both comprehensive and spatially grounded. Current vision-language models, VLMs, can write fluent and detailed captions, but tying those captions reliably to the pixels they describe is still hard. Methods that combine dense captioning with pixel-level grounding tend to fail one way or the other: the description is incomplete, or the segmentation masks are inaccurate. A caption of that kind is a map of the whole image in words, with every phrase pinned to the region it describes.

The authors study the problem through a task they call panoptic grounded captioning. A VLM must describe both the foreground objects and the background regions of an image, and ground every referring phrase with a pixel-level mask. Panoptic means nothing is left out: things and stuff, objects and surroundings alike.

The work makes three contributions. The first is a way to measure the task. PanoCaps is a human-annotated benchmark built from panoptic segmentation datasets. Its dense captions cover nearly every pixel, and it aligns image and text at the level of individual entities, so it serves for training as well as evaluation. With it come a protocol for matching phrases to masks, and a generalized Panoptic Quality metric, gPQ, that scores agreement in text and in masks together. The point of the metric is to judge a caption and its masks as one output, rather than scoring the words and the pixels separately.

The second contribution is the model. PANORAMA treats phrase grounding as selection: choosing from a pool of mask proposals conditioned on the phrase. The VLM conditions a pretrained segmenter on contextualized representations of each phrase, obtains candidate masks, and learns to select the ones that correspond to that phrase. The segmenter supplies the candidates. The VLM, knowing the phrase in the context of the whole caption, decides which of them the phrase means.

This selection interface is trained jointly with caption generation. That joint training is what lets PANORAMA produce high-quality masks, and it allows each phrase to refer either to a single region or to several instances at once. The second case matters in practice, since a phrase such as a plural noun can refer to many objects at once.

The third contribution is the results. PANORAMA achieves the best overall grounding on PanoCaps, and on several pixel-level grounding tasks it matches or exceeds specialized models. The experiments show precise segmentations at the level of entities, while the captions stay detailed and consistent with their masks. Code, data and models are available.

Casting grounding as a choice among proposals, rather than drawing a mask from scratch, lets a captioning model lean on a pretrained segmenter for pixel accuracy while keeping control over which region each phrase means. The benchmark, the metric and the model together define the task, measure it, and set a first strong result on it.