SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
Yan Zhan, Shaobo Liu, Qiunan Liu, Yuanjun Shi, Siqi Xu, WeiYi Hou, Xiang Xu, Zekang Li and 2 others
Peking University · Shenzhen University · Tencent PCG QQ Team
Published September 24, 2026 · on the feed September 28, 2026 · ▲ 12 · arXiv 2609.29050
The abstract
Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens. Consequently, gradient noise from summary generation leaks into tool-decision tokens, causing cross-segment credit misattribution and brittle optimization. In this work, we propose SLCA-GRPO, a framework incorporating Segment-Locked Credit Assignment (SLCA). To enable scalable exploration without costly real APIs and stable training, we first construct the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure. Building on this, SLCA decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), SLCA routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination (the dominant cross-segment credit misattribution channel) within each policy update. On a 7B backbone, SLCA-GRPO accelerates convergence and outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on τ^2-Bench under the same training budgets, achieving higher accuracy with reduced tool redundancy and costs.
The Brief
2:35A short podcast summary of the paper. Written and voiced by AI.
Transcript
SLCA-GRPO. Resolving Cross-Segment Credit Misattribution in Tool-Calling RL. By Yan Zhan and nine others, from Peking University, Shenzhen University, and one other institution. Published on the twenty-fourth of September. Its code repository is worked out: about nine hundred source files, with a readme and a licence.
A tool-calling agent produces two different kinds of output in one response. There are structured tool invocations, and there are natural-language summaries written for the user, interleaved together. Standard on-policy reinforcement learning does not see that difference. Algorithms like GRPO take a single scalar advantage for the whole trajectory and broadcast it to every token alike. The consequence is that gradient noise from generating the summary leaks into the tokens that decide which tool to call. Credit lands on the wrong segment, and optimization becomes brittle. The authors call this cross-segment credit misattribution.
SLCA-GRPO answers it with Segment-Locked Credit Assignment, SLCA. The core idea is to stop treating the trajectory as one undivided thing. Advantage is estimated separately at the level of each structural segment, so tool tokens and summary tokens are each credited for what they are responsible for.
The framework has three pieces. First comes infrastructure: a Schema-Guided LLM Simulator, SGLS, which allows scalable exploration and stable training without calling costly real APIs. On top of that, SLCA decouples advantage estimation by segment within a single group of rollouts, and it does so without needing additional rollouts from intermediate states. Finally, Hierarchical Rewards, HierR, supply the two signals SLCA routes. Execution advantages go to tool tokens. Preference advantages go to summary tokens. According to the authors, this eliminates advantage contamination, which they identify as the dominant channel of misattribution, within each policy update.
The experiments use a seven billion parameter backbone and hold training budgets equal across methods. SLCA-GRPO converges faster and outperforms standard GRPO, ToolPO, and RLTR. The margin is two point five three percentage points on in-domain evaluation, one point three six points on the Berkeley Function-Calling Leaderboard, and nine point one five points on tau squared bench. It reaches higher accuracy while also cutting tool redundancy and cost.
One reward number for a response that does two jobs is where the trouble started. The fix gives each job its own advantage, drawn from the same group of rollouts, and the gains show up on in-domain tasks and on both outside benchmarks alike.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here, a good summary is covering for bad tool calls.
You are training a tool-calling agent with reinforcement learning. Each response mixes two kinds of output: structured tool calls, then a natural-language summary written for the user.
The road ahead
- 1 · The problem ~7 min When one score for a whole response is spread over every token, the tool calls get blamed or praised for how the summary turned out.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Split each response into a tool part and a summary part, score each, and send each part's advantage only to its own tokens.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Simulate tools, sample a group of rollouts, score each with two rewards, normalize each per segment, and route each advantage to its own tokens.
- 6 · The experiment ~6 min You’ll guess: Under equal training budgets on the 7B backbone, by how many percentage points did SLCA-GRPO outperform matched standard GRPO on tau squared bench?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.