Can Large Language Model Agents Learn by Watching Tutorial Videos?

Can Large Language Model Agents Learn by Watching Tutorial Videos?. By Haoyu Zhao and two others, from University College London. Published on the first of November.

People keep learning from carefully made materials, books and tutorial videos among them, and carry what they learn into later tasks. Millions of instructional videos are uploaded online every day. That is an ever-expanding wealth of human teaching, added to daily, and LLM agents remain unable to exploit it. Whatever an agent knows, it knew on the day its training finished, and the tutorials posted since then pass it by. Their capabilities are frozen once training ends, and updating their parameters at scale remains prohibitively expensive. That leaves a pointed question: can an agent learn from this vast and growing body of human teaching without any parameter updates at all?

The answer proposed here is Tutorial Agentic Learning, or TAL. It is a nonparametric continual-learning paradigm, meaning the knowledge goes somewhere other than the weights. Tutorial videos are consolidated into a persistent, structured memory the authors call a tutorialbook. The model stays fixed. The tutorialbook grows as the agent watches more.

Building the tutorialbook is a two-stage adversarial process. It harvests hard questions grounded in the videos, then uses those questions to refine the memory. Retrieval at inference time also works differently from the usual approach. Rather than matching by embedding similarity, TAL retrieves through a Table-of-Contents tree, a TOC, which ties what gets retrieved directly to the model's own reasoning.

To study this setting, the authors also introduce VidLearn, a multimodal question answering benchmark. It is built from seventy-two chess tutorial videos, spanning twelve openings. Chess makes the setting concrete: whatever the agent comes to know about those openings, it has to take from the videos rather than from any update to its weights.

On VidLearn, TAL outperforms every baseline evaluated, and it improves monotonically as it consumes more videos. Compared with the corresponding vanilla multimodal models, its relative gains reach up to one hundred thirteen percent on the training split. On the test split, they reach up to ninety-one percent. The monotonic part matters for the paradigm: each additional video consumed leaves the agent better off, which is what learning from a growing stream of tutorials would require.

The evaluation also moves from questions to the board itself. In actual gameplay, TAL reaches a stronger opening-phase position than its vanilla counterpart in sixty-nine percent of head-to-head games. An evaluation in a second domain is reported as further evidence that the approach is general.

This relocates where learning lives. Continual learning is usually imagined as a change to the model. Here it is a change to what the model can consult, a structured book written from watching tutorials and organised so that finding the right page is part of reasoning. The authors' conclusion is that instructional videos can serve as a scalable resource for agents that keep learning without updating their parameters.