Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models

Autoregressive Mosaics. Probing 2D Spatial Reasoning in Text-Only Language Models. By Ashwin Nedungadi and two others, from the University of Rostock. Published on the first of September. No code repository is linked to it.

Large language models trained only on text and code can sometimes write programs that draw recognizable images. A model that has never seen a picture produces code, the code runs, and out comes something a person can name. The puzzle is what that ability actually reflects. One reading is that the model holds some internal representation of two-dimensional spatial layout. Another is that it is simply good at translating spatial descriptions into code, and that the picture is a side effect of fluent programming. Looking at a finished drawing cannot tell those two readings apart, because both produce the same output.

Autoregressive Mosaics, called AM-Bench, is built to separate the two factors. It measures them with a pair of tasks that pull in different directions. One isolates translation: can the model turn a fully specified geometry into code. The other isolates layout: can the model decide where things go when nobody has told it. If a model does well on the first and poorly on the second, its drawing ability is not coming from layout at all.

The translation task gives a model the complete geometry of a picture, written out in words, as its prompt, and asks for the code that produces it. Nothing has to be invented. Every position is already stated, and the job is faithful conversion. The layout task removes that support. The prompt is underspecified, and the model has to compose the image itself, choosing an arrangement the words only imply. Beyond the two tasks, the study adds an ablation of the output medium, swapping procedural code for raw SVG, and it probes model activations to look for a layout plan inside the network.

Eight open-weight models trained only on text and code were tested. On translation, all of them reliably turn specified geometry into code. On layout, their performance differs substantially. Since they are equally capable at the translation step, those differences are not explained by code-generation ability alone.

The medium turns out to matter as well. Replacing procedural code with raw SVG improves layout scores across every model, so the interface a model expresses itself through shapes how well it lays things out. The probing results are the most specific. A coarse layout plan is present in the activations before generation begins, but it reflects only the layout the prompt implies. Once drawing starts, the models track the evolving geometric state as it changes, rather than carrying out a plan fixed at the outset.

Put together, two-dimensional spatial performance in a text-only model depends on two things: which model it is, and which medium it draws in. Code skill is not the explanation. And the picture of how the drawing happens is incremental. There is a rough sketch in mind at the start, drawn from the prompt, and then the model keeps track of the canvas as it goes, one step at a time.