Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Hongbang Yuan, Zhuoran Jin, Yixin Cao
Fudan University · CASIA · Shanghai Innovation Institute
Published September 8, 2026 · on the feed September 9, 2026 · ▲ 23 · arXiv 2609.08404
The abstract
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional agent-side warming up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to environment-side adaptation by constructing Feedback-Enriched Environments (FEEs). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs (1) stabilizes training dynamics by reducing entropy volatility, (2) facilitates proactive state-space exploration in difficult tasks, (3) ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and (4) identifies intra-group feedback consistency as a critical boundary for stable optimization.
The Brief
2:28A short podcast summary of the paper. Written and voiced by AI.
Transcript
Environments as Scaffold. Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks. By Hongbang Yuan and two others, from Fudan University, the Institute of Automation, Chinese Academy of Sciences, and one other institution. Published on the eighth of September. Its code repository is worked out: about four hundred and fifty source files, with a readme and a licence.
Large Language Models are remarkably good at static reasoning, but training them as autonomous agents with Reinforcement Learning, on long-horizon tasks, is often held back by severe reward sparsity. Success comes rarely, and a rare reward is little to learn from. The usual remedy works on the agent side: warm the agent up with supervised fine-tuning, or SFT, before reinforcement learning begins. That helps, but it is frequently limited by scarce data and by constrained exploration.
The authors propose moving the adaptation to the other side, the environment. They construct Feedback-Enriched Environments, or FEEs. Instead of preparing the agent to cope with a sparse environment, they reshape the environment so it gives the agent more to learn from, acting as a scaffold for an agent that is evolving itself.
A pilot study establishes the feedback design strategy. Environments are reformulated so that their feedback shifts, over time, from action guidance to observation enrichment. That transition happens in the later stages of two processes: exploration within an episode, and evolution across episodes. Early on the environment points toward actions; later it enriches what the agent observes instead.
Large-scale experiments on the SciWorld and BFCL benchmarks, using several scales of Qwen3 and reinforcement-learning algorithms including GRPO, GSPO and DAPO, show FEEs consistently improving on standard settings.
The analysis identifies four effects of training with FEEs. It stabilises training dynamics by reducing volatility in entropy. It encourages proactive exploration of the state space on difficult tasks. It ensures that environmental guidance is internalised into the policy weights, rather than working only as a prior at inference time. And it identifies consistency of feedback within a group as a critical boundary for stable optimisation.
The third effect is the one that justifies the approach. Richer feedback during training could have been a crutch the agent leans on and loses when it is taken away. The analysis indicates the guidance ends up in the weights, so the scaffold can come down and what it taught remains.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here the wall is silence: an environment that almost never says well done.
You are training a language-model agent with reinforcement learning on long tasks: many dependent steps in a simulated lab before any success. Almost every attempt earns zero reward.
The road ahead
- 1 · The problem ~7 min Long agent tasks almost never end in success, so reinforcement learning gets no reward to learn from, and the usual warm-up has real costs.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Leave the agent alone and enrich what the environment tells it: next-step hints early, richer state information later.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Enrich two standard benchmarks with early hints and later state information, then train Qwen3 agents with unchanged RL algorithms.
- 6 · The experiment ~6 min You’ll guess: Across Qwen3 model sizes and RL algorithms such as GRPO, GSPO and DAPO, how did training in Feedback-Enriched Environments compare with standard settings?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.