Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?. By Jialv Zou and eight others, from Huazhong University of Science and Technology, Meituan LongCat, and five other institutions. Published on the twenty-seventh of October.

Discrete visual tokens describe images in a discrete form, much as words describe language. That makes them a native interface for multimodal models trained on one shared objective, predicting the next token. But there is a widely held assumption working against them. Quantization error, the thinking goes, causes semantic loss that cannot be avoided. On that view discrete tokens are fundamentally limited in how much meaning they can carry, and continuous visual representations stand as their built-in upper bound.

The authors revisit that assumption, using multimodal understanding as a proxy for semantic fidelity. Quantization error turns out to be an unreliable guide: discrete representations with substantially larger errors can understand better. If error were what destroyed meaning, more error should always mean worse understanding, and it does not. So error cannot be the whole story of what a discrete code keeps or loses. What emerges instead as a key determinant of downstream performance is the effective bit budget of the discrete indices. There is a catch. Existing quantizers eventually saturate, because each additional codebook contributes less semantic information than the one before. Raising the bit budget is supposed to help, but with those quantizers the extra bits buy less and less, and the curve flattens before discrete tokens catch up with continuous ones.

As one way to ease that saturation, the paper introduces Factorized Vector Quantization. It quantizes complementary views of the visual feature, each of a fixed dimension, using symmetric codebooks that run in parallel. Unlike the existing quantizers, it keeps improving as the bit budget grows, and eventually matches continuous visual representations on multimodal understanding.

The factorized design is the answer to the saturation problem stated just before. Where adding codebooks to an existing quantizer brings diminishing returns, splitting the feature into complementary views and giving each its own parallel codebook lets the bit budget keep paying off. Beyond the quantizer itself, the authors identify key principles and a practical recipe that make the resulting model competitive, though the abstract names the recipe without spelling it out.

Trained on industrial-scale multimodal data, the model reaches state-of-the-art performance among discrete-representation models. On a nine-metric average it outperforms UniAR by twenty-two point eight percent.

Against Emu3-Chat, on the same average, the margin is fifty-nine point two percent. To the authors' knowledge it is also the first discrete-representation model to be competitive with strong continuous-representation models such as Qwen3-VL, and that includes fine-grained OCR tasks.

The title asks whether discrete tokens can be semantically lossless. The answer offered here is that the limit people blamed on quantization error was really a limit of bit budget, and of quantizers that stop gaining from it. Lift that, and the authors see discrete visual representations as a plausible foundation for large-scale native multimodal models.

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?. By Jialv Zou and eight others, from Huazhong University of Science and Technology, Meituan LongCat, and five other institutions. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

Turning an image into discrete tokens, the way text is already tokens, would let one model read pictures and words under a single next-token objective. The assumed obstacle is quantization error. Snapping a continuous visual feature to the nearest entry in a finite codebook throws something away, and the common view is that what it loses is meaning. On that view continuous features are a ceiling that discrete tokens can never reach, and the gap is worst when reading text inside images, where fine detail matters most. The authors set out to test that assumption rather than accept it.

Two ideas carry the argument. The first is the yardstick. Semantic fidelity is measured by multimodal understanding: if a representation lets a language model answer any question about an image, it counts as semantically lossless for that image. The second is the bit budget. Pixels, continuous features and discrete indices are all finite bit strings on a computer, so each has a nominal capacity per image patch. For discrete tokens, that capacity is the number of codebooks times the logarithm of the codebook size. A codebook of sixteen thousand three hundred and eighty-four entries gives fourteen bits per patch. Doubling its size adds a single bit, while adding a second codebook adds fourteen. Scaling by codebook count is the efficient direction.

Once bits matter, the question becomes why existing multi-codebook quantizers stop gaining. Residual quantization hands each later codebook the leftover error of the ones before, so later codebooks see ever smaller residuals. Product quantization splits the feature into channel groups, so each extra codebook sees an ever narrower slice. Either way, nominal bits grow faster than the information each new codebook can actually carry. Factorized Vector Quantization, or FVQ, is built against both failures. Every codebook gets its own projection of the whole feature, dense over all input channels, into a view of fixed size. Adding a codebook adds another view rather than shrinking the existing ones. Each view is quantized independently and in parallel against its own codebook, so no codebook depends on another's output and all play symmetric roles. A patch becomes a tuple of indices, one per codebook.

The projections are not learned. The authors sample a fixed random Gaussian matrix, orthonormalize its columns with a QR decomposition, scale it, and cut its rows into blocks, one per codebook. That construction satisfies a tight-frame condition: taken together, the projections stretch every direction of the feature by exactly the same factor. No direction is lost by all views at once, and none is amplified over another. The practical payoff is that a scaled transpose undoes the projection exactly, so the factorization adds no error of its own. All distortion comes from rounding each view to its nearest codeword. Training then happens in two phases. The tokenizer learns only to reconstruct frozen features from a vision encoder, with no pixel loss at all. Then it is frozen, and the language model receives nothing but the integer indices, turned into embeddings learned from scratch.

The comparison that tests the thesis is the scaling curve, run with a small language model under identical settings. Going from eight codebooks to sixteen, residual quantization gains nothing and product quantization gains two tenths of a point. FVQ gains one point four and reaches sixty-five point three, against sixty-four point six for the original continuous features. The finding also travels to other encoders. With MoonViT-v2 features, eight-codebook FVQ lifts the average from thirty-eight point seven to sixty-one point one over a single codebook. With SigLIP2 it rises from forty-four point seven to fifty-three point nine.

At full scale, sixteen codebooks on the Qwen3 ViT encoder feed Qwen3-8B, trained on industrial-scale data. Among discrete models, its nine-metric average beats UniAR by twenty-two point eight percent. Against Qwen3-VL-8B, it edges ahead on MathVista, posts the best RealWorldQA score in the comparison, and leads on both CharXiv tasks, while landing close on DocVQA and OmniDocBench.

The paper has no limitations section, though it calls FVQ one approach to the saturation rather than the answer. Every table gives single numbers with no error bars or repeated runs, and the margin by which FVQ matches continuous features is under one point. Its decisive advantage over the other quantizers rests on a single step of the curve. Fidelity is measured only through understanding, so the generation uses that motivate discrete tokens are not tested. At full scale the model still trails Qwen3-VL-8B on MMMU-Pro, OCRBench, ChartQA and InfoVQA. Some baseline numbers were rerun by the authors and others taken from earlier papers. And the striking qualitative examples were, in the authors' own words, manually selected and not intended as statistical evidence.

The idea to carry away is a change of question. How much a discrete token distorts the feature turns out to be the wrong measure. The better one is how many informative bits the indices carry, and whether each new codebook still sees something new. Give every codebook a full view, keep the codebooks equal, and make sure the views jointly lose nothing, and discrete tokens start looking like a budget you can spend.