PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latents
Vighnesh Subramaniam, Boris Katz, Brian Cheung, Chun-Liang Li, Tomas Pfister, Yale Song
MIT CSAIL · Google Cloud AI Research · UCSF
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 6 · arXiv 2609.36199
The abstract
Diffusion models can produce striking images and videos, but they still struggle with the compositional details that make a generation faithful to a prompt, such as object counts, attribute binding, spatial relations, and temporally grounded actions. A common way to improve prompt satisfaction is to spend more compute at test time through Best-of-N sampling, but final-sample selection is fixed. Best-of-N can only choose among completed outputs and cannot repair a promising trajectory before it fails. We introduce PreviewDiff, a training-free test-time search method that turns diffusion sampling from scalar search into a multimodal critic-guided search over intermediate latents. At selected denoising checkpoints, PreviewDiff decodes a partial preview, asks a multimodal judge to score and critique it, and uses the resulting natural-language feedback to branch over semantic prompt edits and locally re-noised latent continuations. These branches are then scored and selectively rolled forward, allowing verifier compute to guide generation while the sample is still editable. Across image and video generation benchmarks, PreviewDiff consistently improves over budget-matched Best-of-N selection and strong scalar-search baselines. Ablations show that earlier interventions and increased search width provide the largest gains, while deeper search and additional semantic variants offer complementary improvements. PreviewDiff demonstrates that multimodal feedback is most useful not only as a final verifier, but as an active controller inside the denoising process.
The Brief
2:36A short podcast summary of the paper. Written and voiced by AI.
Transcript
PreviewDiff. Multimodal Critic-Guided Search over Diffusion Latents. By Vighnesh Subramaniam and five others, from MIT CSAIL, Google Cloud AI Research, and one other institution. Published on the twenty-eighth of September. No code repository is linked to it.
Diffusion models can produce striking images and videos, yet they still stumble on the compositional details that make a generation faithful to its prompt. Object counts. Attribute binding. Spatial relations. Actions that have to be grounded in time. A common remedy is to spend more compute at test time through Best-of-N sampling: generate several outputs and keep the one that satisfies the prompt best. The weakness is when that choice gets made. Final sample selection is fixed. Best-of-N can only choose among completed outputs, and it cannot repair a trajectory that looked promising before it failed. By the time any verifier sees the sample, there is nothing left to steer.
PreviewDiff is a training-free, test-time search method that moves the verifier earlier. Instead of a scalar search over finished samples, it runs a multimodal critic-guided search over intermediate latents. The judge is consulted while the image or video is still being formed, and it contributes more than a score: it contributes a critique, in words, and those words help decide where generation goes next.
At selected denoising checkpoints, PreviewDiff decodes a partial preview of the sample in progress. A multimodal judge scores that preview and critiques it. The resulting natural-language feedback then drives branching in two directions. One direction is semantic edits to the prompt. The other is continuations of the latent that have been locally re-noised. These branches are scored in turn, and only some of them are selectively rolled forward. The effect is that verifier compute guides generation while the sample is still editable, rather than ranking samples that can no longer change.
Across image and video generation benchmarks, PreviewDiff consistently improves over Best-of-N selection at a matched budget, and over strong scalar-search baselines. The ablations show where the gains come from. Intervening earlier, and widening the search, provide the largest improvements. Searching deeper, and adding more semantic variants, offer further gains, and those are described as complementary to the first two.
A judge that only grades finished work can pick a winner but cannot coach one. PreviewDiff's argument is that multimodal feedback is most useful not only as a final verifier, but as an active controller inside the denoising process. The ablations lean the same way: the earlier the critique arrives, the more of the sample is still open to change, and the larger the payoff from listening to it.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A diffusion model can paint almost anything, then quietly get the count wrong.
You generate images from prompts like three red apples left of a blue cup. The results look striking, but they miscount objects, swap colours, or put things in the wrong place.
The road ahead
- 1 · The problem ~7 min Image generators look great but get prompt details wrong, and picking the best finished sample comes too late to repair an early flaw.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Consult a multimodal judge on rough previews during denoising, turn its written critique into new branches, and keep only the best-scoring ones.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min At each checkpoint: decode a preview, have the judge score and critique it, branch on its fixes, then score branches and keep the best.
- 6 · The experiment ~6 min You’ll guess: In PreviewDiff's ablations, which changes to the search provided the largest gains?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.