Foundations of Large Language Models

Foundations of Large Language Models. By Tong Xiao and one other. Published on the eighth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

This one is not a research paper in the usual sense. It is a book, and its subject is large language models. The title states its intention plainly. The authors are after foundations: the concepts underneath the field, rather than a complete tour of every cutting-edge technology that has appeared on top of them. That is a deliberate choice of depth over breadth, and the abstract is candid that comprehensive coverage is not the aim.

The material is arranged into six main chapters, and each one takes up a key area of its own. The first is pre-training. The second covers generative models. Then come prompting, and alignment. The fifth chapter turns to inference, and the sixth to reasoning. Read in that order, the list traces a path from how a model is first built, through how it is asked to do things and brought into line with what people want, to how it is run and how it is made to work through problems.

The book names its readers directly. It is written for college students, for professionals, and for practitioners in natural language processing and the fields around it. Beyond those groups, the authors suggest it can serve as a reference for anyone with an interest in large language models. That spread of audience fits the decision to stay with foundations, since concepts are what a student and a working practitioner share.

What the abstract offers, then, is a map rather than a claim. Six areas, one focus on the concepts that hold them up, and an open invitation to anyone who wants a grounding in how these models are made and used.