Beyond Selection: Token Parameterization for Extreme Visual Token Compression

Beyond Selection. Token Parameterization for Extreme Visual Token Compression. By Rui Zhong and three others, from Zhejiang University. Published on the twenty-eighth of September. Its code repository is public and real: seventeen source files.

Compressing visual tokens makes vision-language models more efficient, but under extreme compression budgets both standard routes run into trouble. Squeezing an image down to a handful of tokens forces a choice about what survives. Pruning tokens can break visual grounding. Learned resamplers, the other route, increase parameter count, attention cost and training complexity.

The authors step back and look at compression as a question of how tokens are parameterized. They separate two parts. One is a change of basis followed by structured truncation, which decides what subspace is kept and how compressible the input is. The other is how the coordinates are organized, which governs optimization and alignment across modalities. That split yields two coupled objectives, compressibility and learnability, formalized as unified functionals. Compressibility asks how much of the image survives the cut. Learnability asks whether what survives is arranged so that training and alignment with text go smoothly.

Braco is the coder designed under those objectives. It is lightweight and works in four steps: truncation in a transform basis, basis-coordinate embeddings that do not depend on the input, an orthogonal re-parameterization that depends on the budget, and learned spatial residual tokens produced by lightweight pooling.

Braco sets the favourable accuracy-efficiency frontier between twenty-three times and sixty-four times compression, and stays competitive at one hundred forty-four times. There it reaches ninety-five point two percent accuracy while cutting prefill compute by roughly eighty-four to eighty-seven percent against the uncompressed upper bound. Against prior methods it matches or improves accuracy, with up to about thirty-six percent end-to-end speedup. The compressor itself is far cheaper than those alternatives, with latency lower by sixteen point six times and compute lower by seventy-eight point eight times.

The title says beyond selection, and that is the shift. Instead of choosing which tokens survive, Braco changes the basis they are written in, truncates there, and then attends to how the remaining coordinates are organized so the model can learn from them.