Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
Nanjing University · Institute of Automation, Chinese Academy of Sciences
Published November 9, 2026 · on the feed November 10, 2026 · ▲ 39 · arXiv 2611.12886
The abstract
Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.
The Brief
2:15A short podcast summary of the paper. Written and voiced by AI.
Transcript
Reframe. Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning. By Haozhi Yuan and three others, from Nanjing University and the Institute of Automation, Chinese Academy of Sciences. Published on the ninth of November.
Agentic reinforcement learning and multi-teacher on-policy distillation, known as MOPD, offer complementary routes to post-training. For visual agents that use tools, though, both run into a pair of related challenges. A model may overlook useful observations that its tools return. And poorly grounded actions can limit the evidence available for the distillation that follows. In the first case the tools supply something the model ignores. In the second, the tools never supply it at all.
Reframe is a post-training framework that treats the observation state as an object of alignment, rather than only the output distribution. The target is not just what the model says, but what the model has in view when it says it.
The first stage is Ground-Drop-GRPO, or GD-GRPO. It trains spatial and temporal specialists with evidence-aware rewards, and it stochastically removes the original visual input. That combination encourages the specialists to acquire grounded evidence and to use the observations their tools return. With the original input sometimes missing, what the tools bring back becomes something the specialist has to rely on rather than something it can skip past.
The second stage is Observation-Gated-MOPD, or OG-MOPD. Partway through a rollout, it replaces deficient tool actions with canonical ones, then resumes distillation from the repaired observation. The two stages complement each other: a policy that has learned to rely on tool-returned observations is better placed to benefit when those observations are repaired. Neither stage requires a task-specific SFT cold start.
Extensive experiments cover fourteen image and video benchmarks, and across them Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO. In other words, the first stage does what it was built for: the specialists lean on what their tools return more than a model trained only on outcomes does.
Matching a teacher's outputs is only part of what Reframe aligns. It also asks whether the student had the right evidence in front of it, and when a tool action falls short, it repairs that evidence mid-rollout and carries on.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Calling a tool is easy. Looking at what it returns is another matter.
You are training a vision-language model that uses tools: it crops images or selects video frames to find fine-grained evidence. But calling a tool does not mean the model uses what comes back.
The road ahead
- 1 · The problem ~7 min A visual model that calls tools can ignore what they return, and a badly aimed tool hands later teachers a view with no evidence.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Align what the model has in view, not just what it says: reward grounded tool calls, hide the original input, and repair bad calls during distillation.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train grounded specialists with GD-GRPO, then distill them into one student, repairing its deficient tool actions mid-rollout before teacher feedback continues.
- 6 · The experiment ~6 min You’ll guess: Across 14 image and video benchmarks, how did Reframe compare with Mix-RL and vanilla MOPD?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.