On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer

On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer. By Zhenyi Shen and ten others, from King's College London, Tencent Youtu Lab, and one other institution. Published on the fourth of November.

Picture two language models working as a pair. A larger one reads the prompt and builds its representations of it; a smaller one takes those representations and does the generating, which is the cheaper part to run. That arrangement is called Prefill-to-Autoregressive Cross-model Transfer, or PACT, a form of latent communication between models, and it promises a balance between capability and inference cost. The claim at the heart of this work is a caution about it. When a PACT system improves on a task, that improvement alone does not establish that the receiving model benefited from anything the sender told it. Some of the gain can come from somewhere else entirely.

To test this, the authors build PLatour, a PACT framework that connects frozen models through lightweight linear bridges. The bridges start at zero, are trained directly with task rewards, and the framework supports hybrid-attention architectures. On its own terms PLatour does well. It significantly outperforms open-source baselines on generation tasks, and on shared mathematics and coding benchmarks it surpasses the results Mostik reported, while using a substantially smaller sender.

Then come the controlled experiments. On some tasks, the same gains can be reproduced when the sender's signal is uninformative, when a model communicates with itself, or when only the receiver is trained. The conclusion the authors draw is that a trainable bridge can behave as a parameter-efficient adaptation mechanism: it improves the receiver without exploiting useful information from the sender at all.

The finding is stated for some tasks, not all of them, so it does not say that PACT never works. What it says is that a score improvement is not, by itself, evidence of communication. To separate the two, the authors propose PACT-Check, a measure of how much of a trained PACT system's gain genuinely comes from communication. The paper closes with a discussion of the opportunities and challenges PACT systems face.

One habit of mind sits under all of this. Before crediting the message, check whether the listener, trained on its own, would have done just as well. For PACT, that check now has a name and a procedure, and the strong headline numbers from PLatour come paired with the controls needed to read them honestly.