Mosaic: Multi-Session Agentic Reinforcement Learning

Mosaic. Multi-Session Agentic Reinforcement Learning. By Yilong Xu and five others, from the Hong Kong University of Science and Technology, Guangzhou, Huawei, and two other institutions. Published on the fifteenth of October. Its code repository is worked out: about three hundred source files, with a readme and a licence.

Most agentic reinforcement learning methods assume a trajectory lives in a single session: one context that keeps accumulating as the interaction goes on. Agents at inference time do not always work that way. Context compaction and subagent forking can both produce trajectories that span several sessions. Training under a single-session assumption and then executing across many sessions is a mismatch. The first move is to widen the setting. Multi-Session Agentic Reinforcement Learning is defined to allow an arbitrary number of sessions inside one trajectory, with the single-session case as just one instance of it.

Within that setting, a widely adopted approach is to compute an advantage for the whole trajectory and broadcast it to every session. That treats all sessions alike. It cannot tell which session contributed what, and it gives no guidance on how sessions should be allocated within a trajectory. Mosaic replaces broadcasting with critic-free hierarchical optimization, which assigns credit at two levels instead of one.

At the trajectory level, Mosaic incorporates session-usage signals into advantage estimation, so how a trajectory spent its sessions becomes part of how it is judged. At the session level, each session's advantage is shaped according to its estimated marginal contribution. Sessions inside the same trajectory no longer have to share one undifferentiated score. Neither level depends on a learned critic.

Experiments run on BrowseComp with Qwen3.5-4B, in both scenarios that motivated the setting, context compaction and subagent forking. In each, Mosaic outperforms broadcasting, and it does so while using fewer sessions on average. Further analyses report improvements in session utility and in session allocation: more is gained from each session, and sessions are handed out more sparingly.

Part of the contribution is a reframing. Once a trajectory can hold many sessions, a single shared advantage leaves two questions unanswered: which session earned the result, and how many sessions a task should get. Mosaic addresses both inside one objective. The authors point to session utility and allocation as a promising target for future advances in multi-session agentic reinforcement learning.