Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning
Haoqiang Kang, Yinpeng Chen, Luyang Liu, Jesper Sparre Andersen, Abhijit Ogale, Baochen Sun, Lichan Hong, Ed H. Chi
Google DeepMind · UC San Diego
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 5 · arXiv 2608.19669
The abstract
Latent reasoning has advanced multimodal reasoning through a two-stage training paradigm: (1) a helper image is encoded into latent tokens to teach visual chain-of-thought during a supervised fine-tuning (SFT) stage, and (2) these latent tokens are further refined with reward feedback during a reinforcement learning (RL) stage. In this paper, we identify two key limitations of this framework, one in each stage. First, the SFT stage typically relies on an off-the-shelf vision encoder to encode the helper image, yielding suboptimal latent representations that may not be well aligned with the downstream reasoning task. Second, existing RL methods treat the latent component only through deterministic regularization, which constrains policy drift but does not create alternative latent trajectories for exploration. To address these limitations, we propose Scaffolding Minds. Our approach learns a dedicated scaffolding encoder that provides an optimized target in latent space, and learns both the mean and variance of the RL sampler. We further show that these two improvements are complementary, together yielding substantial gains over strong baselines. Empirically, our method improves over the strongest latent reasoning baseline by +9.5 points on FrozenLake spatial planning, with the gain widening to +19 points on the 32x32 grids, and by +5.6 points on average across nine visual-centric reasoning benchmarks.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
Scaffolding Minds. Optimizing Latent Visual Target Representations for Multimodal Reasoning. By Haoqiang Kang and seven others, from Google DeepMind and UC San Diego. Published on the twenty-ninth of September. No code repository is linked to it.
Latent reasoning has pushed multimodal reasoning forward through a two-stage recipe. In the supervised fine-tuning stage, a helper image is encoded into latent tokens, and those tokens teach the model a visual chain of thought. In the reinforcement learning stage, the same latent tokens are refined further using reward feedback. The authors find a limitation in each stage.
In the first, the helper image is usually encoded by an off-the-shelf vision encoder. The latent representations that come out are suboptimal, and may not be well aligned with the reasoning task they are supposed to serve. In the second, existing methods handle the latent component only through deterministic regularization. That keeps the policy from drifting too far, but it never creates alternative latent trajectories for the model to explore. One stage aims at the wrong target, and the other gives no room to search.
Scaffolding Minds answers each limitation directly. For the fine-tuning stage, it learns a dedicated scaffolding encoder, whose job is to provide an optimized target in latent space, rather than whatever a generic encoder happens to produce. For the reinforcement learning stage, it learns both the mean and the variance of the sampler, so the latent part of the policy is no longer pinned in place.
The two changes are meant to work together, and the authors show that they are complementary: combined, they yield substantial gains over strong baselines. One fixes what the model is taught to aim at while it is being fine-tuned. The other gives the model something to explore once rewards take over. A better target with no exploration, or exploration around a poor target, would each leave half of the diagnosis unanswered.
Measured against the strongest latent reasoning baseline, the method gains nine point five points on FrozenLake spatial planning. On the largest grids, thirty-two by thirty-two, that gain widens to nineteen points. Across nine visual-centric reasoning benchmarks, the average improvement is five point six points.
Read as a whole, the work treats the latent tokens as something to be designed rather than inherited. Their target is learned instead of borrowed from a stock encoder, and their sampling is learned instead of fixed. The clearest sign in the numbers is on FrozenLake, where the margin roughly doubles once the grids reach their largest size.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A model that thinks in pictures is only as good as what it is taught to aim at.
Your vision-language model reasons by slipping small blocks of latent visual tokens into its chain of thought. In training, a helper image, such as an annotated crop, is encoded to give those tokens a target.
The road ahead
- 1 · The problem ~7 min Latent visual tokens learn to copy a target made by a general-purpose encoder, so they may aim at features the reasoning task does not need.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Train the encoder that makes the latent target through the answer loss, then let reinforcement learning sample and explore latent actions around it.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train a scaffolding encoder for the target, fine-tune the model to produce that target alone, then refine latent tokens with reward and learned Gaussian sampling.
- 6 · The experiment ~6 min You’ll guess: By how many points did Scaffolding Minds beat the strongest latent reasoning baseline on FrozenLake spatial planning?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.