Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement

Generalized Agent Iteration. One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement. By Hongyao Tang and three others, from Tianjin University and Shanxi University. Published on the eleventh of September. No code repository is linked to it.

When people speak of recursive self-improvement, or RSI, the paper asks what exactly they mean. Is it a phenomenon, a mechanism, or a prospect? RSI is being claimed at many scales, on the road toward autonomous and evolving intelligence, yet no single framework formally describes these emerging instances. Its counterpart in the classical realm, iterative policy improvement, does have one: generalized policy iteration, or GPI. That framework applies broadly and has well-understood theoretical properties. But GPI covers only the case where the update principle and the evaluation base lie outside the agent. The rule for changing the policy, and the yardstick for judging it, are both supplied from elsewhere.

The paper proposes Generalized Agent Iteration, or GAI, a formal framework in which iterative policy improvement and RSI are two cases of one learning paradigm, rather than two separate subjects. GAI defines the agent as a configuration of modifiable components within a system. It models learning as a cycle of two steps: agent evaluation, then agent improvement, then evaluation again.

Two pivotal dials then distinguish one instance from another. The first asks whether the mechanism doing the improving is part of the agent. The second asks whether the standard the agent is measured against is grounded outside it. The first dial draws the boundary between GPI and RSI. When the improver sits outside the agent, the classical account applies. When it sits inside, the system is improving itself, and so, potentially, improving the thing that does the improving.

The second dial sets what the paper calls a system's polarity, which takes one of three values: anchored, goal drift, or fully self-referential. An anchored system is measured against something outside itself. At the far end, a fully self-referential system supplies its own standard, and goal drift names the territory between. With both dials in hand, existing systems can be placed as coordinates on the same two axes. The defects of recursive self-improvement then become statable one condition at a time, rather than as a single vague worry about systems that change themselves.

The framework rests deliberately on the classical account. Its reach depends on accepting the agent as a configuration of modifiable components, and learning as an evaluate and improve cycle. The authors present it as a first step toward a formal characterization of RSI, not a finished theory.

What the paper offers is a vocabulary with coordinates. If GPI and RSI are the same loop with the improver and the yardstick moved to different places, then claims about self-improving systems become comparable, and the question of whether a system's standard can drift becomes one that can be asked precisely. The stated aim is a principled basis for analysing existing systems and designing new ones.