Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Doohyuk Jang, Yoonsik Park, Gyouk Chu, Sihwan Park, Eunho Yang
KAIST · AITRICS
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 59 · arXiv 2609.37868
The abstract
Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.
The Brief
2:12A short podcast summary of the paper. Written and voiced by AI.
Transcript
Learning Beyond What You Sample. Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR. By Doohyuk Jang and four others, from KAIST and AITRICS. Published on the twenty-ninth of September. No code repository is linked to it.
Reinforcement learning with verifiable rewards, in methods such as GRPO, learns from the model's own successful trajectories. With a finite rollout budget, a group of attempts on a prompt can fail every time. An all-fail group carries no reward-based policy-gradient signal at all, and the prompt teaches nothing. Sampling more rollouts raises the chance of a success, but at a higher cost.
The observation behind this work is that a success missing from one model's rollouts may already have been found by another model. Heterogeneous models often succeed on complementary prompts. That opens a door to mutual learning, with no designated stronger teacher: each model can supply what the other missed. GRAFT, short for Gated Replacement of Answer-Failed groups with peer Trajectories, is built to use that complementarity. When a model's group fails entirely, GRAFT replaces it with an informative group from the peer.
What is transferred is the peer's whole group, successful and unsuccessful responses together, along with advantages that the peer computed. Because those trajectories come from a different model, they are off-policy for the learner, and GRAFT controls the mismatch at two levels. At the level of the sequence, a compatibility weight scales each borrowed response. At the level of the token, importance ratios are clipped.
Tests cover three heterogeneous model pairs and five mathematical reasoning benchmarks. With the same per-model rollout budget as GRPO, GRAFT consistently improves both models in each pair, by two point one points on average and by up to four point five points in model-level average performance. The peer does not even need to be training at the same time. Using stored peer trajectories instead of simultaneous co-training keeps most of the gain: one point eight points over GRPO on average.
Think of two students who get stuck on different homework problems. Neither needs to be the better student for each to learn from the other's worked answers, as long as someone accounts for the fact that the answers were written in a different hand. GRAFT turns that into a training rule, and it costs no extra rollouts.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here, prompts where every attempt fails and nothing is learned.
You train a small model on maths problems. For each prompt it samples eight answers and learns from how each one scores against the others.
The road ahead
- 1 · The problem ~7 min When every sampled answer to a prompt fails, a group-based learner gets no signal from that prompt, and buying more samples is costly.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min When your model's group fails completely, borrow the peer model's whole mixed group on that prompt, with its advantages, and limit the mismatch.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Both models sample groups, all-fail groups are found, the peer's mixed group replaces them, and the receiver updates with weights and clipping.
- 6 · The experiment ~6 min You’ll guess: If the peer model is not training alongside, and GRAFT reuses the peer's stored trajectories instead, what happens to its gain over GRPO?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.