Copy-Paste Language Models

Copy-Paste Language Models. By Nathan Godey and one other, from Cornell University. Published on the thirteenth of October. No code repository is linked to it.

Transformers are intuitively considered efficient at copying from their inputs. The authors challenge that intuition, not by showing that copying fails, but by showing it can be done better. The ability of large language models to copy in context, they report, improves when copying is modelled explicitly.

Copy-Paste Language Models, or CPLMs, add a lightweight Copy-Paste head alongside the conventional language modeling head. Copying gets its own dedicated route instead of being left entirely to the usual head.

The new head sits next to the usual one rather than replacing it, and the authors frame its value through what it changes. CPLMs improve training efficiency and downstream performance, and they offer new operating points in long-context and retrieval setups. The overheads, in training and at inference, are described as negligible.

The validation is done by pretraining models from one hundred and thirty-five million parameters up to one point seven billion. Improvements appear at all scales and across backbones. The explanation offered is about gradients: CPLMs partially mitigate the language modeling head's gradient bottleneck by providing an alternative gradient path.

Two things are on offer here: a small addition to the model, and a stated reason it helps. The addition is a second head for copying. The reason the authors give is not about copying as such but about gradients, a second path by which learning signal can reach the model.