Latent Compaction with Test-Time Training

Latent Compaction with Test-Time Training. By Yingfa Chen and three others, from Tsinghua University. Published on the fourteenth of October. Its code repository is public and real: about thirty source files.

Long contexts make key-value caches large, and a large cache raises both memory use and attention cost during generation. Compressing the cache is the obvious response, but it carries a hard requirement. The compressed cache has to preserve information for questions nobody has asked yet. Whatever is thrown away cannot be chosen with the eventual question in mind.

Latent Compaction learns a compact cache specific to one document, using test-time training. The cache itself becomes the thing being trained. The model's weights stay frozen throughout. Only the cache entries change.

Initialisation draws on three signals: reconstruction importance, teacher attention, and diversity in key space. Training then runs in two phases. First, with the entries chosen by reconstruction held fixed, the cache is fitted so that its attention outputs and log-normalizers match those of the full cache. Second, the entire cache is optimised jointly, through end-to-end distillation from the model running on the full cache. Throughout, training uses only generic prompts about the document. It has no access to downstream questions or their answers, which is what the requirement of preparing for unseen questions demands.

Experiments on RULER, LongBench, and Phonebook Lookup show substantial gains over compression baselines held to the same budget. One example from RULER: at thirty-two times compression, the method reaches ninety-six point seven percent accuracy. KVZip plus reaches forty percent at the same setting. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. Code is released.

The authors draw a reframing from these results. Treat the key-value cache as a neural network, optimise it with test-time training against the model that holds the full cache, and retrieval accuracy can survive compression that is aggressive. Nothing about the model changes. What gets learned is a smaller memory of one document, prepared without knowing what will be asked of it.