Amortizing Sequential Experimental Design in Reasoning LLMs
Anita Kriz, Antonin Berthon, Nicolás Astorga, Mihaela van der Schaar
University of Cambridge
Published October 14, 2026 · on the feed October 29, 2026 · ▲ 9 · arXiv 2610.20832
The abstract
Despite their extensive pretraining knowledge and strong reasoning capabilities, large language models (LLMs) struggle to adaptively gather information in experimental design tasks. We study how to amortize sequential Bayesian experimental design (BED) into an LLM by explicitly training the reasoning that guides action selection using information-gain rewards. Our approach specifies the experimental generative model independently of the LLM's own beliefs, allowing us to optimize expected information gain under a model of our choosing. We investigate the design choices for amortization, including alternative generative-model constructions, reward estimation, hypothesis-space specification, multi-step objectives, and computational costs relative to training- and inference-time BED baselines. On Twenty Questions, the resulting policies improve information acquisition and task performance, outperform inference- and training-time BED baselines at substantially lower computational cost, and generalize to unseen categories. We further demonstrate positive transfer from Twenty Questions training to Medi-Q, a real-world clinical dialogue benchmark. These results suggest that information gathering can be learned as a generalizable reasoning skill in LLMs.
The Brief
1:54A short podcast summary of the paper. Written and voiced by AI.
Transcript
Amortizing Sequential Experimental Design in Reasoning LLMs. By Anita Kriz and three others, from the University of Cambridge. Published on the fourteenth of October.
Large language models arrive with broad pretraining knowledge and strong reasoning, yet they struggle with a particular kind of task: gathering information adaptively. In experimental design, the job is to choose the next question or test so that its answer tells you as much as possible, then update and choose again. Knowing a lot does not automatically make a model good at deciding what to ask.
The authors study how to amortize sequential Bayesian experimental design, or BED, into an LLM. Rather than running an expensive design procedure every time a question must be picked, they train the reasoning that guides action selection directly, using rewards based on information gain. A key choice is that the experimental generative model is specified independently of the LLM's own beliefs. That lets them optimize expected information gain under a model they choose, not one the LLM happens to hold.
Much of the work is a study of the design choices this involves. They compare alternative ways of constructing the generative model, ways of estimating the reward, how the space of hypotheses is specified, and objectives that look several steps ahead. They also weigh computational cost against BED baselines that operate at training time and at inference time.
The testbed is Twenty Questions. There, the trained policies improve both information acquisition and task performance. They outperform the inference-time and training-time BED baselines at substantially lower computational cost, and they generalize to categories not seen in training. Training on Twenty Questions also transfers positively to Medi-Q, a real-world clinical dialogue benchmark.
The move from a guessing game to clinical dialogue is what gives the result its reach. If a skill learned on one carries to the other, the authors suggest, then information gathering is something an LLM can learn as a reasoning skill that generalizes, rather than something bolted on at query time.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Knowing a lot is not the same as knowing what to ask. That is the wall here.
You want a language model to play a guessing game well: ask questions whose answers narrow down the hidden item fastest. It knows a lot and reasons well, yet it asks poorly chosen questions.
The road ahead
- 1 · The problem ~7 min Language models know a lot, yet they choose poor questions when they must gather information turn by turn and adapt to each answer.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Train the model's reasoning to pick questions, rewarding each question by its expected information gain under a world model the researchers specify.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Fix a world model, score each question by expected information gain, train the reasoning with reinforcement learning, then test on a clinical benchmark.
- 6 · The experiment ~6 min You’ll guess: On Twenty Questions, how did the trained reasoning policies compare with inference-time and training-time BED baselines?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.