Co-Evolving Acting and Forecasting for Language Agents

Co-Evolving Acting and Forecasting for Language Agents. By Haofei Yu and eight others, from the University of Illinois Urbana-Champaign, the University of California San Diego, and one other institution. Published on the fifteenth of October. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

Training language agents is held back by the cost of long-horizon rollouts. Yet standard agentic reinforcement learning learns from one scalar reward at the end of a trajectory, and the supervision sitting in the intermediate steps goes largely unused. Prior work has tried to recover it in two ways. Value targets estimate a per-step probability of success. Observation targets fit what the environment returns. The authors argue that both are hard to construct well. Value targets all derive from the same single scalar signal, repeated for every step in the trajectory. Observation targets, such as web pages and terminal logs, are long and noisy, and they reach only one step ahead.

CoPE adds a third kind of target: multi-step actions. On successful on-policy trajectories, the policy backbone is trained to generate the sequence of actions that follows each history. Because every history gets this treatment, any given action is supervised not only at its own step but also when it is forecast from the histories that came before it. The authors describe the result as a temporal ensemble, which the shared backbone absorbs during training. Acting and forecasting live in one model and learn from the same targets, and that is the sense in which they co-evolve.

The forecasting exists only in training. At inference the policy still acts one step at a time, so there is no overhead. There is also nothing extra to deploy: a single policy backbone does both jobs, with no additional models and no separate pretraining. The supervision comes from action sequences the agent itself produced on trajectories that succeeded, which avoids the two complaints raised against the earlier targets. Actions are not one scalar copied to every step, and they are not pages of environment output.

Five benchmarks are used: ALFWorld, ScienceWorld, WebShop, tau-squared bench, and AppWorld. Across the simulated environments, CoPE achieves stronger aggregate performance than GRPO, and than reinforcement learning baselines that use predictive auxiliary targets. On both realistic applications, it improves overall performance.

Sample efficiency is reported on ScienceWorld. There CoPE delivers up to one point nine times higher sample efficiency than ECHO, and up to two times higher than GRPO. The authors stress that these gains come with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is public.

Underneath the method is a reframing of what a successful trajectory can teach. A terminal reward says only that the whole attempt worked. The actions inside a success, read forward from every point along the way, say what working looked like at each step. CoPE turns that into supervision for the same backbone that acts, and leaves untouched how the agent behaves once it is deployed.