A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

A Glance Is All You Need. Single-Pass Fine-Grained Image Captioning with SimLoss. By Suryaansh Jain and ten others, from the University of Massachusetts Amherst and Adobe Research. Published on the first of September. No code repository is linked to it.

An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models write fluent, high-level captions, yet they routinely miss the details that make an image visually specific: attributes, counts, textures, materials, and spatial relations. Recent multi-stage systems win back some of those details. They generate, decompose, verify, and rewrite. But every extra stage costs time, and the price is substantially higher latency at inference.

SimLoss is a training objective that aims for fine-grained captions in a single pass. It works in embedding space and needs no reference captions. The vision-language model is trained so that a projected version of its hidden state lines up with a frozen embedding of the image. That gives the model a dense signal of visual supervision before any text is decoded at all. The supervision lands on the representation, not on the words. Nothing in the setup adds a stage at inference time. The captioner still reads the image and writes its caption in one go.

The alignment is enforced with an InfoNCE contrastive loss. Because the target is an image embedding rather than a caption, the method needs neither fine-grained captions written by humans nor pseudo-captions produced by a multi-stage pipeline. It comes in two variants. SimLoss FFT backpropagates through an embedding model that is available locally, so the whole thing is fully differentiable. SimLoss GRPO instead treats that embedding model as a black box and uses it as a reward.

The comparison covers four kinds of baseline: single-pass captioners, multi-stage verification, reward-optimized methods, and perception-aware methods. SimLoss FFT, the fully differentiable variant, achieves the highest precision of the group. It nearly matches the F1 score of the multi-stage method, while keeping single-pass inference and running roughly twenty times faster than the multi-stage pipeline.

SimLoss GRPO, the variant that learns from a reward, attains the strongest recall. The two variants therefore lead on different sides of the same measure. One is the most precise, saying the fewest things that are not there. The other has the best recall, leaving out the fewest things that are. Neither of them gives up the single pass to get there.

The claim is about where the detail has to come from. Multi-stage systems recover it at inference, by checking and rewriting. SimLoss moves that work into training, by supervising the model's internal representation against the image itself. The results suggest that embedding-space supervision can recover the quality of multi-stage verification at the latency of a captioner that looks once.