Xiaomi-Lens-0 Technical Report

Xiaomi-Lens-0 Technical Report. By Pei Fu and fourteen others, from Xiaomi. Published on the sixteenth of October. Its code repository is public and real: nineteen source files.

Multimodal large language models answer questions about whole images well. Ask about a particular region, though, and they still struggle with that fine-grained understanding. The report points to one key reason. Text-only supervision never specifies which regions of the image support an answer. A model trained only on words is told what the answer is, but never told where in the picture the answer came from.

Xiaomi-Lens-0 is a four billion parameter multimodal model, built on Qwen3.5, that predicts masks from its own visual representations. The point of that phrase is ownership. Mask supervision is routed into the same backbone that does the answering, rather than handed off to a separate segmentation encoder that the language model never sees inside.

A lightweight decoder combines patch features from the vision encoder with what the report calls visual latents, which are the language model's mask-token states. Because the decoder reads those states, mask supervision flows back into the shared backbone, with no additional segmentation encoder in the way. A second consequence follows. Since each latent decodes to a mask, the model can interleave its reasoning with region-specific latents. It thinks, in effect, with visual latents. Grounded reasoning traces then supervise the regions the model visits during its chain of thought, so the where is trained alongside the what.

The headline numbers come from referring tasks. Across all eight splits of RefCOCO, RefCOCO+ and RefCOCOg, Xiaomi-Lens-0 gains five point six points on average over Qwen3.5 in referring comprehension. It also reaches the highest segmentation cIoU among the models compared. Beyond referring, it surpasses Qwen3.5 across all of the perception benchmarks reported, which matters because Qwen3.5 is the very model it was built on.

Read as a whole, the report makes one move and follows it through. If the missing ingredient is supervision that says where, then let the model's own tokens decode to masks, and let those masks enter both its reasoning and its training at the same time.

Xiaomi-Lens-0 Technical Report. By Pei Fu and fourteen others, from Xiaomi. Published on the sixteenth of October. This is the long version: the whole paper rather than the abstract.

Multimodal models handle questions about whole images well, but they miscount nearby objects and misread the attributes of small regions. The SeerRay team's diagnosis is about supervision. When a model learns only from text answers, the image is just something to condition on. Nothing in the training signal says which pixels support the answer, or how far the relevant region extends. A segmentation mask says exactly that. The question the report sets itself is whether dense mask supervision, trained into the model's own visual representations, makes it perceive better even when all it outputs is text.

Two terms first. Referring expression comprehension asks for a box around whatever a phrase describes. Referring expression segmentation asks for a pixel mask. Models that already learn masks, such as LISA and Sa2VA, mostly take their spatial detail from an extra segmentation encoder, so the model's own visual features never have to encode region extent. A second line, PixelLM and MLLMSeg, decodes masks from the model's own vision features, but keeps the vision encoder frozen, so the mask loss never reaches it. And both lines judge mask supervision by segmentation and grounding scores, not by whether perception in general improves.

Xiaomi-Lens-0 is a four-billion-parameter model built on Qwen3.5. Its vision encoder produces a grid of patch features, and a spatial merger then groups neighbouring patches before the language model sees them. The mask decoder taps the grid before that merger, because the merger's aggressive downsampling discards fine detail, while residual connections carry early fine-grained signals through to the final layer.

To name a target, the model writes a special mask token. The language model's hidden state at that token is called a visual latent. Its state depends on the image, the instruction and everything said so far, so it stands for the region under discussion. The decoder refines the patch grid with one transformer layer and projects it into a shared space. The projected latent then acts as a classifier over patch locations, scoring each patch by how well it matches. The scores are laid back out as a grid and upsampled by two transposed convolutions into a full mask. Patch features are computed once per image; each new latent is simply a new query.

Training adds a mask loss, pixel-wise cross-entropy plus a Dice term for overlap, to the ordinary next-token loss. Stage one freezes the backbone and trains only the decoder. Stage two unfreezes everything, so the mask loss flows into the encoder through the patch features and into the language model through the latent.

Because any mask token can be decoded, a reasoning trace can place one beside every region it mentions: one per frame when counting twenty-two picture frames, one per object in a spatial comparison. The authors call this thinking with visual latents. Its training traces come from a reasoning model, kept only when an independent checker has reproduced the answer, with a mask attached to each box reference. One detail tempers the name. At inference the masks are an output branch only. No crops or region features are fed back to the language model, so the masks make the reasoning inspectable; they do not let the model look again.

On the spatial tasks the gains are broad. Box localization beats the Qwen3.5 backbone on all eight RefCOCO splits, by five point six points on average, and most on RefCOCO+, at seven point six. That dataset bans absolute location words, so targets must be told apart by appearance. At the strict overlap threshold of point nine five, gains over the backbone run from fifteen point five to sixteen point eight points, so boxes are placed more precisely, not just chosen better. Its segmentation score is the best on every split, averaging eighty-one point four against seventy-eight point six for Qwen3-VL-Seg, the strongest baseline. On perception, answered in text alone, it beats its backbone on all eight benchmarks, with counting accuracy on CountQA rising from forty-three point eight to fifty point six.

The comparison that actually tests the thesis is an ablation. A box-only variant is trained on the same stage-two data with the mask loss switched off. Box supervision alone already delivers most of the improvement. Adding masks on top raises all six measured scores again: box localization by between one point one and two point four points, general perception by between point two and one point four. A second ablation removes the grounded reasoning traces, and finds they add little to box scores but sharpen final masks by up to point eight.

The report has no limitations section. Its conclusion names one weakness, and its own benchmark exposes it. LensBench, built by the authors, holds twelve hundred human-reviewed units across four kinds of imagery, pairing explicit instructions that name a category with implicit ones that describe function instead. Xiaomi-Lens-0 leads mask scores on three domains, and Qwen3.8-Max leads on general imagery. On implicit instructions its mask score drops to fifty-four point seven, fifth among the compared models, and for multiple targets in panoramic scenes, box precision falls from eighty-one point three to forty-eight point seven when the category is not named. Two further gaps are visible in what is reported. The ablations measure perception only on three general benchmarks, so the counting gain is never isolated to masks. And no error bars appear anywhere, though the main results average three checkpoints.

The idea to carry away is that a mask can be a training signal for seeing, not just an output format. By decoding masks from the same features and states that produce language, the model is told precisely where its evidence lives, and its text answers improve a little. The latents make a chain of thought checkable region by region, though the model never looks at those regions again. Explicit targets are largely solved here; inferring a set of targets from what they are for is not.