S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang and 13 others
ByteDance Seed · M-A-P · TokenWave.AI
Published August 31, 2026 · on the feed September 3, 2026 · ▲ 29 · arXiv 2608.31100
The abstract
Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce S\textsuperscript{3Gym}, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: Self-Testing, Self-Judging, and Self-Improvement. S^3Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S^3Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.
The Brief
2:40A short podcast summary of the paper. Written and voiced by AI.
Transcript
S3Gym. Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?. By Jiajun Shi and twenty others, from ByteDance Seed, M-A-P, and one other institution. Published on the thirty-first of August. No code repository is linked to it.
Large language models increasingly interact with external environments, and in doing so they accumulate a substantial amount of behavioral experience. Yet existing agent benchmarks largely evaluate them as fixed policies: a model is measured as it is, not as it might become. That leaves an open question. Can an agent actively test its own behavior, judge the experience that results, and then use that experience to make better decisions later on.
S3Gym is an interactive benchmark for exactly that question. It evaluates self-improvement through three capabilities that are coupled together. Self-Testing, where the agent probes its own behavior. Self-Judging, where it assesses what came of it. And Self-Improvement, where it turns that judgment into better decisions.
The protocol separates two phases. Exploration is permissive, so the agent can try things freely. Evaluation is strict and held out, so improvement has to show up somewhere the agent did not practise. This protocol is instantiated in seven text-based games, each with an executable environment verifier. Three pathways for using interaction experience are compared. The first is direct History ICL, which places raw history in context. The second is Summary Memory, conditioned on scores. The third is parameter Training, which writes the experience into the weights.
Self-improvement, the experiments show, is neither automatic nor uniform. Experience at the level of context improves performance for several pairings of model and game. Which pathway works best, though, depends strongly on how the task is structured. Summaries help when experience can be compressed into reusable strategic rules. When success depends on precise information tied to the exact state, summaries often underperform the raw history.
Parameter training cuts both ways. On some tasks it produces substantial gains. On others the improvement is unstable, and there is severe negative transfer. The common thread is that recognizing which actions succeeded is not enough. An agent also has to transform feedback into policies that are executable and that transfer.
The benchmark is built less to crown a winner than to locate where the chain breaks. Testing, judging and improving are separate links, and an agent can manage the first two while failing the third. S3Gym gives a single framework for diagnosing that process, and for finding the bottlenecks that stop interaction experience from becoming reliable self-improvement.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Agents gather heaps of experience. Few benchmarks measure whether they learn from it.
You are asked whether a language-model agent gets better from its own experience. It plays games, piles up trajectories of wins and losses, and in principle could learn from them.
The road ahead
- 1 · The problem ~6 min Agents pile up experience while they act, but today's benchmarks mostly score how good a model is right now, not whether it learns.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Measure self-improvement by letting an agent test itself, judge its experience, reuse it, and then checking it on harder settings it never practised.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min One cycle: explore loose game settings, judge the experience, fold it in through history, summaries or training, then test on strict held-out settings.
- 6 · The experiment ~6 min You’ll guess: Across seven text games, did summarising past experience into rules beat simply feeding the raw history back to the agent?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.