Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Stuck on Shuffle. On Attribute-Object Binding in Vision-Language Models. By Netanel Y. Tamir and six others, from OriginAI, NVIDIA, and one other institution. Published on the fifteenth of October. Its code repository is public and real: ten source files.

Retrieval is a fundamental task in multimodal systems. Given an image or a text query, find the most relevant images or texts in a collection, often by comparing them in a shared embedding space. For that to be reliable, the embeddings have to capture not only which concepts are present, but how those concepts are composed. One key compositional challenge is attribute-object binding: encoding which attributes belong to which objects. A red cup beside a blue plate is not the same scene as a blue cup beside a red plate, even though every word is shared. Existing benchmarks give a limited view of this. They typically cover only a subset of the modality pairings, or simple swaps between two objects, or small sets of attributes. That leaves it unclear how robustly current models hold on to bindings across pairings and in more complex settings.

SABER is a controlled benchmark built to fill that gap. It evaluates attribute-object binding across all four image and text modality pairings: text against text, text against image, image against text, and image against image. It spans diverse attribute types, and its scenes contain up to ten objects, well beyond a two-object swap.

Control is the point of the design. Because the benchmark is built to isolate binding, a model that knows which concepts are present but not which goes with which has nowhere to hide. Running all four pairings means a weakness that only appears in one direction of retrieval is visible rather than averaged away.

Evaluating models on SABER reveals substantial binding failures, in both CLIP-style dual encoders and recent MLLM-based embedders. That raises the next question: how to improve binding efficiently. Multimodal supervision may be effective, but suitable images are difficult and costly to collect. Text, the authors argue, is well suited to binding supervision. It is cheap to generate, it supports a broader range of attributes, and it allows precisely controlled hard negatives.

So they finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. No image supervision is involved. Even so, the gains transfer to cross-modal retrieval and to image-image retrieval, while general multimodal embedding performance is preserved. The resulting model achieves SOTA results on SABER, and it also improves on negation and instance-retrieval benchmarks, which the authors take as evidence that binding-focused supervision has broader benefits.

There are two contributions here, and the second depends on the first. The benchmark shows that today's embedders, of both families, often know what is in a scene without knowing what belongs to what. The fix then shows that the missing skill can be taught through text alone and still carries over to images, which makes binding a cheaper problem to work on than collecting images would suggest.

Stuck on Shuffle. On Attribute-Object Binding in Vision-Language Models. By Netanel Y. Tamir and six others, from OriginAI, NVIDIA, and one other institution. Published on the fifteenth of October. This is the long version: the whole paper rather than the abstract.

Picture a query for a golden cannon, a tie-dye submarine and a glowing giraffe. A retrieval system should return that scene, and not one with a glowing cannon and a golden submarine. The objects are the same, the attributes are the same, and only the pairing differs. That pairing is called attribute-object binding, and the paper's starting complaint is that vision-language embeddings often behave like bags of words: they capture which concepts are present without reliably keeping track of which attribute belongs to which object. Existing tests show only part of the picture: many check a single direction, many use few attributes, often just colour, and most swap attributes between only two objects.

The first contribution is a benchmark called SABER, which stands for Synthetic Attribute Binding Evaluation for Representations. It draws on two hundred and thirty-two everyday objects and fifty visual attributes, spanning colours, materials, patterns and effects, and builds scenes of two up to ten objects. Every question has a query and four candidates, each existing both as an image and as a caption. One candidate keeps the bindings. The other three use exactly the same objects and attributes but reassign them, by permuting the attributes, the objects, or both. Recognising objects is no help: only binding separates right from wrong. And because every item has both forms, SABER tests all four directions, text to text, text to image, image to text and image to image.

Images come from GPT-Image-2, with each attribute rendered the same way across a question's five images, and every image checked by hand. Half the questions form nested chains, each step adding one binding while keeping the earlier ones. That supports a second protocol: retrieval against the whole gallery of seven thousand two hundred candidates, where a larger scene containing every queried binding also counts as correct.

The second contribution is a remedy that uses no images at all. Text, the authors argue, is cheap to generate, makes bindings explicit rather than implicit in pixels, and allows precisely controlled hard negatives. Using Gemma 4, they generate one hundred thousand examples. Each has a short query naming two or three attribute-object pairs, a longer positive sentence that uses those pairs in context, and two hard negatives made by permuting either the attributes or the objects. The negatives contain exactly the same words as the positive. Only the bindings differ. That yields two hundred thousand triplets.

The sentences come in seven construction types, chosen so that adjacency cannot be the shortcut. In some, the attribute is separated from its noun by a verb, as in the book was dated. In others, both nouns come first and the attributes arrive later, through phrases like the former and the latter, or the word respectively. Both of those deferred types are held out of training entirely, to test generalisation. A triplet loss penalises the model whenever the negative sits within a small margin of the positive in cosine similarity. Only small low-rank adapters in the language decoder are trained; the vision tower stays frozen. Transfer comes from this: in a multimodal language model, unlike a dual encoder such as CLIP, images and text pass through shared parameters, so updating them from text also changes how images are embedded. Finally, LiNeS rescales the updates layer by layer to limit forgetting.

Among the pretrained models, the larger Qwen3-VL-Embedding model is strongest, with a mean of fifty-two point five percent on the four-way questions. After text-only finetuning, that rises to eighty-four point two. Text to text becomes nearly perfect. The test of the thesis is the cross-modal directions, because no images were seen in training: text to image gains thirty-four point one points, and image to text thirty-seven point eight. Across nine existing binding benchmarks, the finetuned model beats every baseline in all twelve settings, by fifteen point four points on average over its base.

Three appendix results sharpen the claim. Training only on examples sharing no words at all with SABER still produces the large gains, so this is not vocabulary leakage. On SABER's own data, training on text transfers better across modalities than training on images, by thirteen point one points in text to image retrieval. And an attention analysis finds the base model does bind attributes to objects in early layers, but that signal fades toward zero in the final third of the network. Finetuning preserves it through layers eighteen to twenty-six. The gains even reach negation: the biggest improvement is on pairs like bowl but no cup against cup but no bowl, and it survives removing every training example that mentions negation.

There is no dedicated limitations section. The discussion names one key limitation: SABER is synthetic, and generated images may vary in appearance, layout, object size or orientation beyond the intended binding change. In an appendix, the authors add that the text-versus-image comparison covers one small synthetic benchmark and is not proof that text supervision is generally better. Two problems remain open. On the deferred respectively construction, held out of training, the gain is small, from fifty point eight to fifty-six point six against a chance level of fifty. And in nested retrieval, candidates with only some of the queried bindings outrank valid larger scenes in thirty-three to thirty-nine percent of comparisons. Recognising a superset, which the authors call containment, is still unsolved. Finally, results are reported as single numbers without error bars.

The idea to carry away is that compositional precision is its own axis of embedding quality, separate from broad retrieval skill. The more portable lesson is about where supervision can come from. In a model whose parameters are shared across modalities, a capability can be taught in the cheapest, most controllable medium, plain synthetic sentences, and still show up when the model looks at pictures.