T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang
Tencent Hy Foundation Model Frontier · National University of Singapore · University of Georgia · Indiana University · University of Maryland, College Park
Published September 10, 2026 · ▲ 62 · arXiv 2609.11042
The abstract
Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.
The Brief
2:35A short podcast summary of the paper. Written and voiced by AI.
Transcript
T1. Terminal Agent Reinforcement Learning for Long-Horizon Tasks. By Junyao Yang and six others, from Tencent Hy Foundation Model Frontier, the National University of Singapore, and three other institutions. Published on the tenth of September. No code repository is linked to it.
Agents are increasingly used for long-horizon work such as coding and scientific discovery, and within that, tasks carried out in a terminal are especially important. Training an agent to work through hundreds of shell commands on a single task, with its reward coming from whether the task's own checks pass, is the problem this work takes on. Over that many turns, small instabilities in training have room to compound.
T1 is a Mixture-of-Experts model with one hundred and twenty-two billion parameters in total, trained with reinforcement learning. It operates a real shell in a cloud sandbox, for more than three hundred tool-call turns per task, and it is rewarded by running each task's own verifier. The contribution is presented as a comprehensive recipe in three parts.
First, an aggressive warm start stabilizes actor-critic training, together with a dense process reward that scores a trajectory by the absolute number of verifiers it passes.
Second, optimization is kept stable in two ways. TITO construction trains on the exact token identifiers that were sampled, with drift repaired at turn boundaries. Rollout routing replay, R3, records which experts the sampler chose for each token at every MoE layer, and replays those choices during training. Together, TITO and R3 cut the gap in log-probability between training and inference from zero point zero two one to zero point zero one three, with token drift held at exactly zero in the region the loss covers.
Third, the training corpus is fully out of distribution. Isolated seeds and synthesized tasks are kept disjoint from Terminal-Bench 2.1, so that gains on it reflect real transfer of capability rather than overfitting to the benchmark.
On Terminal-Bench 2.1, the post-training pipeline raises the base model from forty-three point eight percent of tasks resolved to sixty-four percent for T1. On Long-Horizon Terminal Bench, T1 reaches twenty-seven point nine percent, surpassing GPT-5.4 and GLM-5.1.
Each part of the recipe answers a different threat to training over hundreds of turns: a slow start, a mismatch between the model that samples and the model that learns, and gains that might only be memorised. Keeping the training data apart from the benchmark is what lets the improvement be read as capability, including on the long-horizon benchmark, where T1 finishes ahead of GPT-5.4 and GLM-5.1.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Hundreds of turns, one verdict at the end. That is too little to learn from.
You are training an agent with reinforcement learning to work a real shell for hundreds of turns per task. Its reward comes from running each task's own checks.
The road ahead
- 1 · The problem ~6 min Training a shell agent with reinforcement learning is costly, and a pass-or-fail reward returns almost no signal for each expensive run.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Reward the absolute number of passing checks on one fixed scale, and stabilize the critic with an aggressive warm start.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build disjoint tasks, run the agent in a real shell, execute verifiers, count passing checks, then update critic and actor on exact tokens.
- 6 · The experiment ~6 min You’ll guess: Starting from its base model, what share of Terminal-Bench 2.1 tasks did T1 resolve after the post-training pipeline?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.