Rethinking Sequence Compression with Test-time Regression

Rethinking Sequence Compression with Test-time Regression. By Sihan Yang and six others, from the Chinese University of Hong Kong, Independent Researcher, and one other institution. Published on the twenty-second of October. Its code repository is worked out: about five thousand source files, with a readme.

Efficient language and multimodal models keep having to squeeze a long sequence of tokens into a shorter one. That sequence compression is a fundamental operation, and it is also a critical information bottleneck: what gets lost there is lost for everything downstream. The prevailing approaches, based on pooling or on learned projections, are in the authors' assessment still underperforming.

The paper revisits compression through the lens of test-time regression. The input tokens define a regression problem, and the fitted coefficients of that problem become the compressed representation. Instead of averaging tokens or projecting them, the compressor asks what compact set of coefficients best explains them. Whatever the tokens contain, the compressed version is whatever fits them best.

Two things happen at different times. The regression problem itself is learned end-to-end by backpropagation during training. Its coefficients, on the other hand, are computed efficiently in closed form during the forward pass. So training shapes what kind of fit is being asked for, while each new input gets its own fit, solved directly rather than by iteration. The pairing keeps the compressor learnable without making it slow. The result is TTR-Compressor, offered as a general alternative to conventional pooling and projection-based compression.

The authors test it in two roles, one in language and one in vision. As the compressor for block-wise sparse attention, it improves RULER performance over attention-based compressors by a large margin. As the visual token merger for multimodal large language models, it consistently outperforms the standard MLP merger, and matches that merger's performance while using fifty-five point six percent fewer visual tokens.

The authors conclude that test-time regression is an effective primitive for sequence compression. The reframing is the contribution: a compressed sequence is treated not as a summary of tokens but as the solution to a small fitting problem posed by them, solved fresh on every pass.