Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

Continual Learning Exam. Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents. By Haoyu Zhao and eighteen others. Published on the twenty-fifth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Continual learning, for an LLM agent, means getting better over time: acquiring, refining and applying knowledge from experience, while keeping the useful knowledge it already had. Agents that work over long time horizons need exactly that. Yet current evaluations mostly score individual tasks one at a time. They ask whether an agent can do something, not whether it improves because of what it has already done.

Continual Learning Exam, or CLE, is built to measure the second thing: improvement driven by experience, inside realistic professional workflows. It was developed by more than fifty domain experts. The quantity of interest is how far an agent climbs when each step depends on what it learned on the steps below. Single-task scores cannot show that, since an agent could do each task well in isolation and still learn nothing from one to the next.

The structure is a ladder. CLE has one hundred and twenty human-authored tasks, and each is organised as five levels, which gives six hundred levels in total. Those span eighty-nine professional roles across twelve occupational domains. An agent must pass a level to unlock the next one, and later levels build on the experience gathered during earlier assignments. An evaluation agent grades every submission against human-authored rubrics.

Two settings are compared. In one, the agent gets a single submission per level. In the other, a failure brings diagnostic feedback and one chance to revise. Comparing the two isolates a narrow kind of learning: whether an agent that is told what went wrong can use that information to get past a level it had just failed. Across the ladder as a whole, the design tests the broader kind, whether experience on the lower rungs carries upward.

Seventeen harness-model configurations were tested, all built on frontier LLMs. Feedback and revision help: the median gain in completion is five point three percentage points. But the ceiling is low. The best-performing system clears only forty-seven point seven percent of levels, and it completes all five levels in just twenty-three point three percent of tasks. The authors take this as a sign of how hard sustained learning from experience still is for today's frontier LLMs.

Beyond grading, the authors note the dataset can also serve as training material, for post-training LLMs to strengthen their continual learning capabilities. As an exam, though, its verdict is plain. Agents gain a little when told what went wrong, and even the best system reaches the top rung in fewer than a quarter of its tasks.