Tools Need Not Speak in Text: Latent Context Compression for Agents

Tools Need Not Speak in Text. Latent Context Compression for Agents. By Yuetong Liu and four others, from Tsinghua University. Published on the fourth of November.

Agents built on large language models usually append raw tool outputs to their context. Across many steps of interaction, those observations are long and noisy. They add substantial context overhead, and they dilute the agent's attention to the information that matters for the task. Existing remedies, token pruning and summarisation, remain bound to discrete text representations, and that limits how far they can compress.

The authors challenge the assumption that an agent has to consume tool observations as explicit text tokens at all. Their model, Coca-Cola, performs continuous latent context compression for agents. It compresses each tool observation into latent tokens and embeds them directly into the input embedding space of the agent. The observation reaches the agent without ever being spelled out as words. Freed from the vocabulary of discrete tokens, the compressed form is no longer limited by how few words can still say what the tool returned.

Compression alone does not guarantee the agent keeps behaving well. To address that, the paper introduces Action-Preserving Training. It aligns the agent's distribution over next actions under the full observation with its distribution under the compressed one. Training against that target pushes the compression model toward latent tokens that are action-sufficient, holding on to the information that decisions depend on.

Experiments use web-search agents on four QA benchmarks, across two model families. Coca-Cola reduces the agent's token consumption by between seventy point zero one percent and eighty-nine point two four percent. It cuts end-to-end runtime by between twenty-four point four five percent and fifty-five point nine seven percent. And it does this while consistently improving task performance, rather than trading accuracy for savings. The authors present this as a promising approach to context compression in agents.

A tool's output only has to be good enough to choose the next action, and Action-Preserving Training turns that observation into the training objective. Measuring compression by whether the agent would act the same, rather than by whether the text survives, is what lets the context shrink this far without the task suffering.