Competing Learning in Hybrid Architectures

Competing Learning in Hybrid Architectures. By Xingyu Shen and five others, from Tsinghua University. Published on the nineteenth of October. Its code repository is worked out: about sixty source files, with a readme.

Many modern language models are hybrids, combining linear-time recurrence with softmax attention. Modern recurrence layers are a highly expressive mixing mechanism, and they keep throughput high by compressing context into a fixed-size state, but they underperform softmax attention at retrieving information from context. The paper reports a phenomenon it calls Competing Learning. Different kinds of mixers compete to learn the same function, and once any layer has learned a function, it suppresses the learning signal for that function in the other layers. Inside a hybrid, attention and recurrence are rivals for the same job.

Layer attribution analysis shows how the rivalry plays out for retrieval. Under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention does. Beaten to the function, attention then fails to learn retrieval in most cases, even though it is the mixer that retrieves better. An intervention points the same way: stopping weight updates in the recurrence in the middle of training helps attention learn retrieval.

From these results comes a staged training procedure for pre-training hybrid architectures on natural language. On models of up to one billion parameters, it brings substantial improvement in language modeling loss, in common-sense reasoning, and across a wide range of retrieval-related tasks at different context lengths.

Three bounds frame the claim. The models tested reach one billion parameters, and the abstract reports nothing beyond that size. The retrieval result is stated for typical data distributions and for most cases, not for every one. And the staged procedure is described for pre-training on natural language, the setting in which it was evaluated.

The finding changes what a hybrid's attention layers can be expected to do. Attention is the stronger retriever, so one might assume that placing it in the model is enough for it to take on retrieval. Competing Learning says otherwise: if the recurrence learns the function first, attention may never receive the signal to learn it, and the training schedule has to leave it room.