SolveEdit: Benchmarking Visual Problem Solving in Generative Models
Wenjie Shu, Yexin Liu, Harold Haodong Chen, Xuerui Qiu, Zehan Wang, Yidi Zhang, Yizhan Chen, Zunwei Wang and 4 others
ZODA · HKUST · UCAS · ZJU · UTokyo
Published September 28, 2026 · on the feed September 29, 2026 · ▲ 15 · arXiv 2609.35504
The abstract
Machine intelligence is often evaluated through abstract reasoning problems, yet many real-world problems are visual, such as arranging objects, repairing layouts, or tracing routes. Solving these problems requires understanding a scene, inferring what must change to achieve a goal, and realizing that change without disturbing unrelated content. However, existing benchmarks mainly evaluate perception, generation, or explicitly specified transformations, leaving goal-driven visual problem solving underexplored. To bridge this gap, we introduce SolveEpIT, a benchmark for visual problem solving through scene transformation. Given an image and a goal, a model must infer a valid transformation from the request, the scene, or a visually expressed rule, then execute it while preserving unrelated content. SoLvEEDrr contains 2,728 cases. Atomic transition contracts specify required and protected conditions, enabling SoLvEScoRE to measure completion and unintended changes without a single reference output. The strongest evaluated model achieves only57.0% SolvEScore. We further introduce SolveEdiT-PLAN, a two-stage visual planner that instantiates the transition before generation. Under matched single-generation evaluation, it improves SoLvEScoRE by 9.1 points on average across three tested generators, including a gain from 57.0% to 71.6% for GPT-Image-2, without modifying the editor.
The Brief
2:33A short podcast summary of the paper. Written and voiced by AI.
Transcript
SolveEdit. Benchmarking Visual Problem Solving in Generative Models. By Wenjie Shu and eleven others, from ZODA, the Hong Kong University of Science and Technology, and three other institutions. Published on the twenty-eighth of September. Its code repository is public but slight: seven source files.
Machine intelligence is often tested on abstract reasoning puzzles, yet many real problems are visual. Arranging objects, repairing a layout, tracing a route. Solving one means understanding the scene, working out what has to change to reach the goal, and then making that change without disturbing anything else. Existing benchmarks mostly evaluate perception, generation, or transformations that are spelled out explicitly. Goal-driven visual problem solving, where the model has to decide the change for itself, has been left underexplored.
SolveEdit frames visual problem solving as scene transformation. A model is given an image and a goal. It must infer a valid transformation, whether from the request, from the scene, or from a rule expressed visually, and then execute it while preserving the unrelated content. Those are two separate demands. The first is inference: nobody tells the model exactly what edit to make, so it has to derive one from the situation. The second is restraint. A change that reaches the goal but disturbs the rest of the scene has not solved the problem cleanly, and the benchmark is built to notice the difference.
The benchmark contains two thousand seven hundred and twenty-eight cases. Each carries an atomic transition contract, which specifies the conditions that are required and the conditions that are protected. Those contracts are what let the metric, SolveScore, measure both completion and unintended changes without needing a single reference output. Rather than comparing an edit against one correct picture, the score asks whether the required conditions now hold and whether the protected ones were left alone.
The strongest model evaluated achieves only fifty-seven percent SolveScore. Alongside the benchmark, the authors introduce SolveEdit-PLAN, a two-stage visual planner that instantiates the transition before any generation happens. Under matched single-generation evaluation it raises SolveScore by nine point one points on average across three tested generators. For GPT-Image-2, the score climbs from fifty-seven percent to seventy-one point six, and the editor itself is not modified.
Two results sit side by side. Current generators leave a large share of goal-driven visual problems unsolved. And part of that shortfall is recovered without touching the generator at all, simply by working out the transition first and only then handing it to the editor.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a test that only checks edits someone already spelled out.
You want to know whether image generators can solve visual problems: swap the water birds for animals that can swim, connect two points with a wire, place a block in the only empty slot.
The road ahead
- 1 · The problem ~7 min Testing whether image generators can solve visual problems is hard, because they must work out the change themselves and many answers can be right.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give each case a contract of required and protected conditions, sort cases by where the change comes from, and plan the change before editing.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min A two-stage planner works out the change, the unchanged editor makes it in one call, and SolveScore checks the contract.
- 6 · The experiment ~6 min You’ll guess: Adding the SolveEdit-PLAN visual planner before generation, without changing the editor, raised SolveScore by how many points on average across three generators?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.