Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks

Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks. By Maximilian Schlegel and eight others, from Google, MIT, and one other institution. Published on the second of October. Its code repository is public but slight: two source files.

A conventional neural network compresses training data of arbitrary size into weight matrices of fixed size. There is another way to look at the same network. Earlier work by Irie and colleagues used a classic duality: any linear layer in a deep network trained by gradient descent can be expressed as linear attention over the training data points. Read that way, the layer is already retrieving from its training set. The same work suggested replacing linear attention with more powerful attention functions. The authors show that this is harder than it sounds. Naively carrying the learning rules from the linear case over to advanced kernels does not lead to principled optimization.

The paper fills that gap. It investigates a general-purpose layer that does not compress at all. During training it stores a new pair of key and value representations for every data point. At inference time it retrieves and recombines those representations through an attention mechanism. The network therefore grows as it sees more data. The authors call the resulting paradigm retrieval-centric deep learning, or RCDL.

What makes RCDL principled is a set of functional gradient-based learning rules developed for kernelized attention layers. Two families of kernel are covered: radial basis function kernels, and softmax-like kernels. These rules replace the borrowed linear-attention rules that the authors showed do not optimize in a principled way once the kernel becomes more powerful. The point of the functional view is that learning happens over the function the layer computes, not over a fixed set of weights, which is the natural frame for a layer whose memory keeps growing with every new data point. The abstract presents this as the step that turns a suggestive perspective into a working paradigm: the duality had pointed toward stronger attention functions, and these rules are what make training with them sound.

Empirically, the authors report promising performance and learning efficiency for RCDL on image classification and on synthetic teacher-student learning tasks.

A second result is conceptual. Replacing linear attention in the dual form of a network with advanced linear-attention variants, namely MesaNet and DeltaNet, yields a formal connection to recently proposed optimizers for conventional, fixed-size networks. The dual view, in other words, points back at ordinary training.

The move here is to take a duality seriously in both directions. Reading a linear layer as attention over its training data suggests a network that keeps the data and grows, and the paper supplies the learning rules that make that growth principled with stronger kernels. Read the other way, the same duality offers a new perspective on how fixed-size networks are optimized.