Multiscale Policy Optimization for Long-Horizon LLM Agents

Multiscale Policy Optimization for Long-Horizon LLM Agents. By Tao Ren and eighteen others, from Peking University, IQuest Research, and four other institutions. Published on the twenty-sixth of October.

Long-horizon LLM agents operate at two scales at once. They interact with their environment through complete assistant turns, yet each action inside a turn is generated token by token. Training such an agent with reinforcement learning means deciding where the credit for an outcome should land. One option is the turn as a whole. Another is each individual token. The two scales pull in different directions.

The authors start from an observation. Critic values vary primarily across turns, rather than within them. That points toward anchoring optimization at the turn, while still allocating credit more finely among the tokens inside it. Multiscale Policy Optimization, or MPO, is built on that principle.

MPO combines two components. The first is Multiscale Advantage Estimation, abbreviated MAE, which handles how credit is assigned. The second is turn-level importance sampling, which governs how the policy is updated. Together, they align credit assignment and policy updates at the same scale, the turn, which is where the critic's values were seen to move.

Extensive experiments span search, tool-use and automation tasks. They use models of two sizes: one with four billion parameters, and one labelled thirty B, A three B. Across these settings, MPO shows consistent gains over strong RL baselines.

The design follows from where the signal lives. If value estimates change mostly from one turn to the next, the turn becomes the natural unit for both deciding credit and updating the policy, with the tokens inside a turn sharing that credit at a finer grain.