Environment Evolution for Terminal Agents

Environment Evolution for Terminal Agents. By Zhiyuan Fan and eleven others, from the Hunyuan Team at Tencent. Published on the third of September. No code repository is linked to it.

Training terminal agents depends on having interactive, verifiable environments, and scaling those is the bottleneck. There is a moving-target problem underneath it. Environments synthesized from scratch become less challenging as frontier models get more capable, so the learning signal they provide shrinks over time even though nothing about them has changed. Recent co-evolution methods address this by iteratively synthesizing environments near the model's learnable frontier, based on weaknesses exposed during rollouts. But leaning on on-policy rollouts limits generalization, and limits the ability to keep supplying signal as the model gets stronger.

Environment evolution takes the difficulty question off-policy. Rather than reading weaknesses from the model's own current rollouts, it incrementally increases environment difficulty independently, and schedules the evolved environments generation by generation across training. The point of doing it that way is continuity: the supply of harder environments does not depend on what the model happens to be failing at right now.

Three evolution directions are derived from the multi-turn learning objective itself, each one a lever that influences how difficult an environment is. That derivation is the part carrying the weight, because it means the directions are not a list of plausible ways to make things harder. They come out of the thing being optimized. Evolution along those directions is then implemented through a loop-engineered multi-agent harness.

Quantitative rollout experiments with Hy4 preview, Claude Opus 5 and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments, which is the claim the method needs to establish first. Effectiveness is then validated on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon reinforcement learning training, improving performance on Terminal-Bench 2.1 by fourteen point four and eighteen percentage points respectively.

The structural argument here is that tying environment difficulty to the current model's failures makes the curriculum a follower. Generating difficulty off-policy lets it lead instead.

Environment Evolution for Terminal Agents. By Zhiyuan Fan and eleven others, from the Hunyuan Team at Tencent. Published on the third of September. This is the long version: the whole paper rather than the abstract.

Reinforcement learning environments have become the bottleneck for training terminal agents, and the bottleneck has a moving part. Environments synthesized from scratch are frequently too easy for current frontier models, which solve them across repeated rollouts. When that happens the environment is discarded for training, because it produces no signal distinguishing a better trajectory from a worse one. The construction cost is wasted and the diversity of what remains shrinks.

The established response is co-evolution: roll the target agent out on seed environments, collect where it fails into a weakness bank, and synthesize new environments around those failures. It keeps environments near the model's capability frontier, which is the right idea. The paper's objection is that it buys this with a dependency. Difficulty is being estimated on-policy, so what you measure is not how hard the environment is. It is where this particular model is weak, which is a property of its training distribution. As the model improves and failures become sparse, the signal thins out.

The authors instead derive difficulty from the multi-turn learning objective itself, and that derivation is the paper's real contribution. Treat an agent's execution not as a flat sequence of actions but as interleaved scenarios and skills: a situation, and the skill applied in it. The negative log-likelihood of such a trajectory then decomposes into exactly three contributors. The number of meaningful solver turns the trajectory requires, which is execution length. How novel each scenario is. And how rare the required skill is within that scenario. The second and third are policy-dependent as written, so the move is to replace the model's own probabilities with those under a reference distribution grounded in broad world knowledge, which a deep-research agent can estimate through web search. That converts model-specific weakness into model-agnostic environment difficulty.

The implementation is a multi-agent harness built as two gated loops. In the first, a Proposer extracts the expected execution sequence from the current environment and edits it along whichever direction is selected this generation: for length it inserts scenario-skill pairs to introduce dependencies, for scenario it swaps a scenario while keeping its paired skill, for skill it swaps the skill. A rubric-based reviewer iterates on that plan until it passes. In the second loop a Modifier applies the change, and every candidate must clear three verifiers running in parallel. An Oracle verifier confirms the reference solution still succeeds in the sandbox. An Invalid-test verifier confirms an empty or no-op solution fails, which is what makes the verification system trustworthy rather than merely present. And a rubrics verifier checks general quality. There is also a mutation control the authors call evolution effort, with three levels restricting an edit to a single scenario-skill pair, a contiguous span, or an unrestricted portion of the sequence.

Rollout experiments with Hy4 preview, Claude Opus 5 and GPT-5.6 Sol show that the evolved environments get consistently harder generation by generation, across different models, despite being synthesized without rolling any of them out. That is the claim the method has to establish first and it is the one most plausibly at risk from going off-policy. Then two hundred step long-horizon training on Qwen3.6-27B and Qwen3.6-35B-A3B improves Terminal-Bench 2.1 by fourteen point four and eighteen points respectively, beating both co-evolution and ensemble baselines. The most telling observation is smaller: evolved environments keep providing challenging training signal even when their seed environments have already been consumed for supervised fine-tuning. The lineage outlives its own starting point.

This paper carries no limitations section, which is itself a thing to notice. The scope statements have to be assembled from elsewhere in the text. Everything is terminal agents; the authors name software-engineering agents and computer-use agents as future work rather than as demonstrated. The reference distribution underpinning the whole off-policy claim is estimated by a deep-research agent through web search, so the model-agnostic difficulty measure is only as grounded as that estimate. And during development, accepted environments went through human-in-the-loop review, with issues that slipped past the rubrics converted into new rubrics, which means the verification quality reported here has human curation folded into it.

The framing to take away is the inversion. Tying environment difficulty to the current model's failures makes your curriculum a follower: it can only ever go where the model has already been and failed. Deriving difficulty from the task structure instead lets the curriculum lead, and lets a lineage keep producing signal after the model has outgrown where it started.