Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL. By Jinhao Dong and nine others, from Xiaomi, Renmin University of China, and two other institutions. Published on the twenty-sixth of September. No code repository is linked to it.

Reinforcement learning for code agents often rewards a trajectory with a simple pass or fail from executable tests. Under Group Relative Policy Optimization, that means every trajectory in a rollout group that passes the tests receives the same advantage. Two solutions that both pass can still differ a great deal. One may be clean and targeted. The other may carry unnecessary changes, or changes outside the scope of the task. Binary test rewards see no difference between them, and so the policy gets no signal that would push it toward the cleaner kind.

GAGAR adds quality-aware credit redistribution to code agent RL. The name spells out the two parts: groupwise agentic grading, and advantage redistribution. The tests still decide what passes, and nothing about verification by execution is given up. On top of that, an agentic grader ranks the passing solutions against each other, and the advantages are reshaped so that more of the credit flows to the better ones.

It builds on dynamic sampling, which retains the groups that contain both passing and failing trajectories. All trajectories from such a group are placed together in a shared workspace. There, an agentic grader trained with supervised fine-tuning inspects them jointly and ranks the candidates that passed. Lower-ranked trajectories are then downweighted, and the advantages of all the passing trajectories are rescaled proportionally so that their total returns to the original sum. Preserving the sum matters. The relative weights set by the quality ranking are kept, while the overall amount of credit for passing stays the same, only shifted toward the higher-quality implementations.

GAGAR is evaluated at industrial scale, starting from pre-RL fine-tuned checkpoints of two models: MiMo-V2.6-Flash, with three hundred ten billion total parameters, and MiMo-V2.6-Pro, with one point zero two trillion. Controlled experiments using only code on the Flash model show better code agent performance, less growth in trajectory length, and more stable training. GAGAR was also applied in large-scale mixed-task reinforcement learning with both Flash and Pro. The abstract reports these outcomes without specific figures. The shorter trajectories fit the motivation: a policy rewarded for targeted implementations has less reason to wander into changes the task never asked for.

Tests answer whether code works. They do not answer whether it is the code you would want to keep. GAGAR leaves the first question to the tests and hands the second to a grader, and redistributing rather than adding reward means the balance between passing and failing is left exactly where it was.

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL. By Jinhao Dong and nine others, from Xiaomi, Renmin University of China, and two other institutions. Published on the twenty-sixth of September. This is the long version: the whole paper rather than the abstract.

Train a coding agent with reinforcement learning and the usual reward is binary. The agent works through a repository, submits a patch, and the tests either pass or they do not. Group Relative Policy Optimization, or GRPO, then compares each attempt with the others sampled for the same task. Here is the gap the authors point at. Every passing attempt in a group gets exactly the same credit. One patch fixes the root cause with a focused change that follows the repository's conventions. Another adds a scattered special case, weakens validation, and rewrites code nobody asked it to touch. Both passed, so both are reinforced equally, and the policy has no signal telling it which one a developer would actually want to merge.

Two pieces of setup matter. First, the advantage. Following Dr. GRPO, each trajectory's advantage is simply its reward minus the group's mean reward, with no division by the spread. If half the group passes, every pass gets plus a half and every failure gets minus a half, and the advantages sum to zero. Second, dynamic sampling, borrowed from DAPO: only groups containing both passes and failures are kept, because a group where everyone passed or everyone failed carries no signal. The question the paper asks lives inside those mixed groups. Among the passes, which deserve more?

The method is called Gagar, for groupwise agentic grading for advantage redistribution, and it has two halves. The first is the grader. For each mixed group, it receives one shared workspace holding the task, the repository, every trajectory, every patch and the test logs. It is an agent, not a single prompt. It starts from a turn-by-turn summary of the rollouts, then reads the parts that need closer inspection, cross-checks patches against the code and logs, and can run targeted checks of its own. Any negative judgement must cite a patch location, a trajectory event or an execution result. Failed attempts sit in the workspace as context, but only passes get ranked.

Before ranking, the grader looks for passes that relied on leaked or external answers. Those are zeroed and treated as failures. The rest are scored from one to five on five criteria: approach suitability, precision, minimality of changes, unintended side effects, and codebase consistency, weighted from thirty percent down to ten. Scores sort candidates into three tiers. The top tier needs at least four on every criterion. The bottom tier catches unrequested rewrites, test-specific workarounds, the lowest approach score, or low scores on both minimality and side effects. Tier and rank become a discount factor: the best keeps full weight, the bottom tier drops to one fifth.

The second half is the redistribution. Multiply each pass's advantage by its factor, then rescale all the passes by one common number, chosen so their total climbs back to exactly what it was. Failures are untouched. Take two passes in a group of four, with factors of one and four tenths. Plain downweighting would leave them a half and a fifth. Rescaling restores the total, giving roughly seven tenths and three tenths. The ratio between them is still the ratio of their factors, and the group still sums to zero. A pass gains credit when its factor beats the passing average and gives some up when it falls below.

The primary study is code-only training of MiMo-V2.6-Flash, a model with three hundred and ten billion total parameters, against a binary-reward baseline. The baseline had to be stopped at step twenty-eight because its DeepSWE pass rate collapsed, from fifty-eight point five percent to fifty point two. At that step Gagar sat at sixty-two point two percent, and used fewer turns: one hundred and eleven point six on average, against one hundred and thirty-two point three. A separate judge, Claude Opus 5, reviewed a sample of thirty DeepSWE tasks blind and gave Gagar's passing solutions an average win rate of sixty-nine point eight percent.

The comparison that tests the thesis most directly is the ablation: downweighting alone against full redistribution. Without restoring the credit, policy entropy climbed from about a third to roughly zero point nine over thirty steps, and training rollouts more than doubled in length. With redistribution, entropy reached only about a half. On DeepSWE at step twenty-eight, downweighting alone scored fifty-six point two percent against sixty-two point two. Same grader, same factors. Only the credit balance differed.

There is no limitations section, so these come from what the evidence covers. The headline mixed-task results, sixty-seven point nine percent for Flash and seventy-one point nine for Pro, have no matching run without Gagar, so they show the method fits industrial training, not how much it adds there. Each evaluation averages three samples per task, and no error bars are reported. The quality judge uses the same five criteria the grader was built around. Two practical facts hide in the method. The grader costs time: an early version on Claude Opus 5 took about two thousand seconds per group, and the trained grader still takes about six hundred, so grading is overlapped with rollouts for other tasks. And the deployed rescaling is capped at one and a half, so when the cap binds, the exact conservation no longer holds.

The idea to carry away is that a reward signal has a budget. Binary tests decide how much credit a group's successes earn. A quality judgement should decide only how that credit is split among them. Spend it and the balance against failures tips, and training drifts. Keep the total and move it between passes, and the policy learns which passing patch a developer would accept.