LatentPress: Context Compression Beyond Text and Vision

LatentPress. Context Compression Beyond Text and Vision. By Zhengze Zhou and one other, from Cornell University and Arizona State University. Published on the first of September. It links to a code repository that is no longer there.

When a long history has to be compressed, it is usually compressed into something a person could read: a text summary, or a rendered image that gets decoded again. That happens even when the only consumer is a language model, which never needed the human-readable step and has to pay to undo it.

LatentPress writes conversational histories and long documents into a third representation. Continuous memory tokens, which a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference at all. The claim is that soft tokens are a practical machine-facing context interface, sitting alongside text and vision rather than beneath them.

A small writer, matched to the reader, does the compression at ratios from four to sixteen times. Only an adapter is trained, between four point two and twenty-six point two million parameters, which is around a tenth of a percent of the decoder. The decoder itself is untouched, which is what makes this an interface rather than a model change.

On a long-term memory benchmark it reaches zero point five zero four accuracy at seven point seven times compression, against zero point four nine zero for the uncompressed evidence, so compressing does not cost accuracy here. Text summaries manage zero point one eight four and OCR-based compression falls from zero point four two six to zero point three one two. On a long-document question benchmark, in-domain writers match or exceed raw-context reading at four to eight times, while sixteen times trails. The speed numbers are the more practical half. Writing takes forty-three milliseconds per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is five to nine times faster than raw context or cached OCR.

Two transfer settings were tested rather than one, moving between different sources and unseen document domains, which is what distinguishes an interface claim from a benchmark result. The question the work leaves open is what breaks above sixteen times, since that is where the reported gains stop.

LatentPress. Context Compression Beyond Text and Vision. By Zhengze Zhou and one other, from Cornell University and Arizona State University. Published on the first of September. This is the long version: the whole paper rather than the abstract.

Assistants and agents accumulate more history than they can afford to reread. A deployment trace holds instructions, dialogue, plans, tool calls, observations and environment feedback, and a later decision usually depends on only a small part of it. The same pressure appears whenever a model reads long documents to answer a question. In both cases the default machine-facing interface is still discrete text: systems retrieve text, summarize text, prune text, or reconstruct text from another modality before a language model can use it.

The observation the paper builds on is that this is a convention rather than a requirement. Text is convenient for people and interoperable across systems, but a model does not need its stored or compressed context to be human-readable. So the question becomes whether long context can be written into a compact continuous representation that a frozen language model reads without first recovering the text. The framing is deliberately narrow, and the authors are careful about it. They separate context use into a write step, mapping text to a compact state, and a read step, supplying that state to a frozen decoder. That abstraction covers conversational histories and long documents alike, and it does not attempt to replace retrieval, reflection, update policies or conflict resolution in a complete memory system. It asks only what representation should cross the boundary between stored context and the model consuming it.

The mechanism is a direct-read soft-token interface. A reader-matched writer reuses two frozen decoder layers together with a small trainable adapter to map text segments into continuous vectors, and those vectors enter the frozen decoder through its input-embedding interface, followed by the question. The decoder then answers directly, with no text reconstruction anywhere. The adapter is between four point two and twenty-six point two million parameters, roughly a tenth of a percent of the decoder, and the decoder itself is never touched. What distinguishes this from the established family of continuous-vector compression is stated up front: earlier approaches train or adapt a reader or encoder at model scale, whereas this leaves the downstream decoder alone entirely.

On the long-term memory benchmark, the interface reaches zero point five zero four accuracy at seven point seven times compression, against zero point four nine zero for uncompressed evidence. Compressing does not cost accuracy there; it slightly helps, which is the sort of result that usually means the compression is also discarding distraction. Text summaries manage zero point one eight four and OCR-based compression falls from zero point four two six to zero point three one two. On long-document questions, in-domain writers match or exceed raw-context reading at four to eight times compression, while sixteen times trails. The speed figures are the more immediately practical half: writing takes forty-three milliseconds per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is five to nine times faster than raw context or cached OCR.

The limitations are unusually well aligned with the claims. The work isolates compression and reading using oracle evidence sessions, which deliberately leaves out integration with retrieval, memory updates and conflict resolution, so this is a component for a memory system rather than a memory system. The central future direction is dynamic compression: the role-based and uniform rates here are hand-specified heuristics, and a learned policy could choose a rate per segment, preserving detail only where it matters. They suggest optimising such a policy against downstream answer reward under a latency or memory budget. What none of this establishes is what breaks above sixteen times compression, which is where the reported gains stop.

The idea to keep is that the human-readability of an intermediate representation is a cost, not a feature, when no human reads it. Everything between a stored context and the model consuming it has been text by default, and the reconstruction step that default imposes is paid on every single read.