On the Limits of Scale: Inverse Scaling in Vision-Language Models

On the Limits of Scale. Inverse Scaling in Vision-Language Models. By Rohit Saxena and three others, from the University of Edinburgh, Imperial College London, and one other institution. Published on the nineteenth of October. Its code repository is public and real: eleven source files.

Scaling laws say that more parameters, more data and more compute generally buy monotonic improvement. At the aggregate level that often holds. This paper presents systematic evidence that vision-language models break it at the subtask level. There are specific question categories on which the larger model performs worse than its smaller sibling inside the same model family. Not a smaller gain: a loss. The question categories are specific, which is the point. The effect is invisible when results are summed up, and it is the summing up that scaling laws are usually read from.

Start with the scale of the measurement. Nineteen models from four distinct families, across eight multimodal benchmarks that make up nine evaluation splits. Each model answers twenty-nine thousand three hundred and eighty-six questions, for five hundred and fifty-eight thousand three hundred and thirty-four model-question evaluations in total.

The analysis compares every model pair within a family, question by question. That turns up three hundred and seventy-five combinations of subtask and model pair that show inverse scaling, covering four thousand three hundred and forty-two unique queries where the smaller model answers correctly and the larger one fails. The patterns replicate across architectures that were developed independently of each other. That replication is what lifts the result above a quirk of one lab's training recipe.

How far does this generalise? Thirteen subtask and dataset pairs show inverse scaling in more than one family, which the authors read as consistent with the phenomenon being driven by the task rather than by any particular model. The most affected domains in their study are multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps that exceed twenty percentage points in several cases. Diagram understanding, by contrast, looks comparatively resistant, so the effect is not spread evenly over everything a vision-language model is asked to do.

Aggregate scores hide this entirely, because the aggregate still goes up. What the per-question comparison exposes is that the gain is not evenly spread, and in places it reverses. The authors draw the practical consequence: the assumption that a bigger model uniformly improves multimodal capability does not survive the subtask view, and model selection should be made task by task.