Exploring Collaboration between a language and a non-language agent

Exploring Collaboration between a language and a non-language agent. By Harini S I and five others, from Adobe Media and Data Science Research, the Indraprastha Institute of Information Technology Delhi, and two other institutions. Published on the second of September. No code repository is linked to it.

Language models are increasingly deployed as orchestrators, coordinating specialized subagents through natural language to solve complex tasks. But in many important domains, game playing and robotics among them, the strongest available agents are not language models at all. Bringing such an agent into the conversation would require verbalization. Its rich, continuous representations would have to be compressed into sparse textual summaries at every step of the interaction. The question is whether that compression is a bottleneck.

Two contributions address it. The first is a testbed, LLAMIA-Bench: six diverse collaborative chess tasks spanning three facets, namely behavioral imitation, state assessment, and explanation in natural language. Each task instantiates a well-established chess problem that neither the language model nor the chess engine can solve alone. The second is an alternative to verbalization, called latent state internalization. Instead of describing the subagent's state in words, it projects the subagent's continuous representations directly into the language model's token stream, as learned state tokens.

Those state tokens are not computed once and left alone. As actions advance the environment, the state is dynamically re-encoded, so the tokens the language model sees keep pace with the position on the board. The comparison at the heart of the study is then direct: the same collaboration, with the engine's state either internalized as tokens or verbalized as text.

The experiments find a consistent verbalization debt. Internalization outperforms verbalized integration, and the gap widens over the course of training. It also persists as the language model grows from four billion to fourteen billion parameters, so scale does not close it.

A single model of fourteen billion parameters, LLAMIA, trained with latent state internalization, matches or exceeds task specialists across every task in the benchmark. It does the same against frontier models, including GPT five point one with tool access. And it generalizes out of distribution, in settings where finetunes built for a specific task collapse.

Talking has a cost, and here it is measured. Asking a chess engine to explain itself in sentences squeezes a rich internal state into a sparse summary, and the resulting gap does not shrink with more training or a bigger model. Letting the language model read the engine's state directly, as tokens of its own, avoids paying that debt in the first place.