The Numerical Linear Algebra of Large Language Models

The Numerical Linear Algebra of Large Language Models. By Abdelkader Baggag and one other, from the Qatar Computing Research Institute, Hamad Bin Khalifa University and the University of Minnesota. Published on the third of October. No code repository is linked to it.

Numerical linear algebra has long been one of the quiet engines of science. It supplies the tools for solving fundamental problems that arise in scientific and engineering applications, and over the decades it has kept evolving to meet whatever each new wave of discovery demanded.

The authors give a history in two steps. Through the fifties and the sixties of the last century, substantial effort went into methods for eigenvalue problems, which were emerging from the rapidly growing field of aerodynamics. That work led to the discovery of the LR and QR algorithms. Later, attention turned to solving sparse linear systems, which were common in applications such as computational aerodynamics.

Now, they argue, another wave of major scientific advance is under way, and numerical linear algebra is once again at the heart of it. This is the machine learning wave, which the authors describe as utterly disruptive in science and engineering. Many machine learning tools, large language models in particular, are grounded in matrix and tensor methods. As the field approaches artificial general intelligence, in the authors' framing, matrix methods will be called on to play an even more significant role.

The survey centers on machine learning techniques, with a particular focus on large language models, and it is organised around two objectives.

The first runs from language models toward the numerical analyst. It sets out to clarify the core concepts behind large language models in a way that is accessible to specialists in numerical methods: people fluent in matrices, factorizations and solvers, but not necessarily in the vocabulary of modern machine learning.

The second runs in the other direction. It examines the key numerical linear algebra concepts that language model techniques actually employ, and it highlights several significant recent contributions that numerical linear algebra has made to the field.

One thread joins the historical sketch to the survey itself. In each era the discipline was pulled forward by a specific application. Aerodynamics brought eigenvalue problems, and with them the LR and QR algorithms. Computational aerodynamics brought sparse linear systems. In the authors' account, large language models are the current pull, and because these models rest on matrix and tensor methods, the connection is structural rather than incidental.

A second thread is the practitioner's position. The authors note that the speed of the present change makes it particularly hard for numerical linear algebra specialists to adapt. The two objectives answer that directly. One lowers the barrier to understanding the models. The other shows practitioners where their own concepts already appear inside them, and where their field has recently contributed.

This is a survey written as a bridge between two communities. Its premise is historical: numerical linear algebra has repeatedly grown by answering the demands of a new scientific wave, from eigenvalue problems in aerodynamics to sparse systems later on. Its aim is to hand numerical specialists both a way into large language models and a map of where their own methods are already at work, at a moment when the field is changing fast enough to make that handover difficult without help.