SIMIT: Self-Improving Vision-Language Models via Imagination at Test-Time

SIMIT. Self-Improving Vision-Language Models via Imagination at Test-Time. By Mehmet Onurcan Kaya and two others, from the Technical University of Denmark, the Pioneer Center for AI, and one other institution. Published on the eleventh of October.

Picture a vision-language model facing a set of test questions with no labels attached, no annotations, and no outside model to lean on. The goal is to do better on exactly that set anyway. Some test-time methods try by reasoning for longer or sampling repeatedly, but the model itself is left unchanged. Others do update the model, using rewards built from majority agreement among several candidate answers, or from the model's own judgment of whether it got things right. The trouble with those rewards is that they can reinforce the mistakes the model already makes. If the model is confidently wrong, agreeing with itself only makes it more so.

SIMIT takes a different route. Before answering an unlabeled test query, the model creates its own training data, specific to that query. It imagines similar problems whose solutions it controls. The key observation is that a model does not need to solve the target query correctly in order to build useful practice examples for it. It only needs to construct problems where it already knows the answer, because it chose the answer first.

That is literally the order of operations. SIMIT synthesizes diverse triplets of question, answer and image, specifying each answer before realizing the image that goes with it. To cover a wide range of multimodal tasks, it combines native image generation with an extensible skill library for structured visuals such as documents, charts, and diagrams.

Quality control comes in layers. Multi-stage verification checks that each image actually supports the answer it was assigned. Difficulty filtering keeps the examples that are informative. Adaptive budgeting then aims the practice where it is needed, allocating more samples to queries the model is less confident about. The resulting data can improve the model in two ways: in context, as SIMIT-ICL, or in its weights, as SIMIT-FT.

Across seventeen diverse benchmarks, SIMIT outperforms existing self-improvement methods. Its best configuration achieves a mean relative gain of seven point two percent over BAGEL-7B. The training-free variant, SIMIT-ICL, which never touches the weights, comes close behind with six point nine eight percent.

The reversal at the heart of this is simple to state. Instead of asking a model to grade its own answers, SIMIT has it write questions whose answers it fixed in advance, so the practice material is correct by construction even when the model's judgment is not. And used purely in context, with no training at all, that material already delivers nearly the full gain.