Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training
Yuanhao Yue, Qianli Ma, Chengyu Wang, Haoting Wang, Lei Shen, Jun Huang
Alibaba Cloud Computing · Shanghai Jiao Tong University · Fudan University · Xi'an Jiaotong University
Published September 14, 2026 · on the feed September 15, 2026 · ▲ 14 · arXiv 2609.15051
The abstract
Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the Exploration Potential Score (EPS), a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7\% relative improvement in-domain and gains of 11.5\% on MathVision and 11.1\% on MMMU-Pro.
The Brief
2:22A short podcast summary of the paper. Written and voiced by AI.
Transcript
Not All Prompts Are Equal. Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training. By Yuanhao Yue and five others, from Alibaba Cloud Computing, Shanghai Jiao Tong University, and two other institutions. Published on the fourteenth of September. No code repository is linked to it.
In online reinforcement learning, training prompts differ substantially in how informative they are for the current policy. Some are already saturated. Others are too difficult to yield reliable learning signals. Under standard training, both kinds still receive an equal rollout budget, so effort goes to prompts that teach the policy little either way. A saturated prompt has little left to teach, and an overly hard one gives no dependable signal, yet each costs as many rollouts as a prompt in between.
The authors propose an exploration-guided prompt scaffolding framework, which adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models. At its centre is the Exploration Potential Score, EPS: a lightweight, rollout-based proxy for how useful a prompt is, derived from KL-regularized policy improvement theory. The score is the signal that decides which prompts need attention.
EPS is computable directly from on-policy rollout statistics, without additional overhead, since those rollouts are produced by training anyway. Prompts it marks as low-utility are not discarded. Instead, a teacher model generates scaffolded rewrites of them, which preserve the original task intent while making subsequent training more informative. That reframes what the teacher is for. Its supervision refines the training data, rather than supplying outputs for the student to imitate. The prompt pool therefore changes during training: which prompts get rewritten depends on how the policy is doing at the time.
Integrated with GRPO on Geo3K and MMK12, the method consistently outperforms the baseline, on in-domain benchmarks and on out-of-distribution ones. In- domain, the relative improvement reaches up to nine point seven percent. Out of distribution, the gains reach eleven point five percent on MathVision and eleven point one percent on MMMU-Pro.
The obvious response to a prompt that teaches nothing would be to drop it. Here it is rewritten instead, and the teacher's role shifts from answering questions to reshaping them into ones the policy can learn from, while the task each question poses stays the same.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a pile of prompts that teach almost nothing.
You are post-training a multimodal model with reinforcement learning on visual maths problems. Every prompt gets the same number of sampled attempts.
The road ahead
- 1 · The problem ~7 min When every training prompt gets the same number of attempts, prompts the model always or never solves waste that effort, and which ones they are keeps shifting.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Score each prompt's usefulness from rollouts training already makes, and have a teacher rewrite low-scoring prompts into scaffolded versions instead of discarding them.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Training runs a loop: sample rollouts, score each prompt with EPS, send low scorers to a teacher for rewriting, then train with GRPO on the refreshed pool.
- 6 · The experiment ~6 min You’ll guess: Out of distribution, how large a gain did the method reach on MathVision over the GRPO baseline?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.