Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

Incentivizing Self-Learning in Large Language Models via Reinforcement Learning. By Lingkai Kong and eight others, from the Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and three other institutions. Published on the twenty-ninth of October.

Language models tackle knowledge-intensive tasks with two resources: what is stored in their parameters, and what retrieval-augmented generation fetches for them. Neither quite fits problem solving in an unfamiliar domain. There, acquiring knowledge and reasoning with it have to be tightly coupled. Looking up facts and aggregating them is not enough, and neither is learning only from a context someone assembled in advance. A student meeting a new subject does not simply look things up. They read, work out what matters, and then use it. That combined skill is what current setups leave out.

The paper calls the missing capability self-learning: autonomously retrieving, studying and applying knowledge from long-form reference material, on the fly. To study it, the authors pose a task built around textbooks. A model faces mathematical exercises and has to work through the textbook's exposition, learn the relevant principles, apply them, and construct proofs. The reference solutions are withheld, so the book is the only teacher. Whatever the model needs to know, it has to find in the chapters and then turn into an argument of its own.

Two pieces make this trainable. A BookNavigator interface supports the study process, modelled on how a person would work through a book. Reinforcement learning with fine-grained rewards then provides the incentive for the model to learn this way. The pairing is the point: an interface for moving through a book the way a human student would, and a training signal that makes studying with it pay off.

The method improves performance on textbooks and subjects both inside and outside the training domain. For Intern-S2-Preview-35B, overall proof accuracy rises from fifty-one point five percent to seventy-nine point five percent. It also beats baselines trained with reinforcement learning that either have no textbook or use fixed retrieval.

The gains carry over to complex long-context benchmarks with different task types. Over the base model, the improvement is one point seven percentage points on BrowseComp-Plus, and two point three points on CL-bench.

The authors read these results as a sign that reinforcement learning can foster a self-learning strategy that is not tied to one domain. A model rewarded for studying a mathematics book picks up a habit that also helps on long-context benchmarks of quite different kinds. The broader suggestion is that teaching a model to study from long-form corpora may strengthen its problem solving on complex, long-context tasks in general, not just the ones it was trained on.