Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement
Hyun-Kurl Jang, Jihun Kim, Kuk-Jin Yoon
KAIST
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 5 · arXiv 2609.34528
The abstract
Open-vocabulary semantic segmentation (OVSS) enables pixel-level prediction over arbitrary text-specified vocabularies and has shown strong generalization on common benchmarks. However, OVSS performance often degrades in specialized domains such as medical imaging, remote sensing, and industrial inspection, where dense pixel-level masks for adaptation are costly to obtain and require domain-specific expertise. We propose a preference-guided adaptation framework that replaces dense mask supervision with binary preferences. We observe that different prompt templates produce systematically different segmentations for the same image, a phenomenon we call prompt disagreement, and we repurpose it as a built-in source of preference supervision. Building on this, we mine localized preference queries from regions of high cross-template uncertainty, and adapt the OVSS model with Region-Localized Preference Optimization (RLPO) together with consistency regularization that stabilizes updates outside the queried region. Across extensive experiments on the MESS benchmark, the proposed method achieves consistent gains across diverse OVSS backbones without any pixel-level annotation, and remains effective under noisy preferences. Our code is available at https://github.com/blue-531/pref-ovss.
The Brief
2:05A short podcast summary of the paper. Written and voiced by AI.
Transcript
Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement. By Hyun-Kurl Jang and two others, from KAIST. Published on the twenty-eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Open-vocabulary semantic segmentation, or OVSS, predicts a label for every pixel from whatever vocabulary a user types, and it generalizes well on common benchmarks. Move it into a specialized domain, though, and performance often slips. Medical imaging, remote sensing and industrial inspection are the examples given. Adapting to those domains would normally mean dense pixel-level masks, and those masks are costly to obtain and need a domain expert to draw them.
The proposal is to replace dense mask supervision with binary preferences: which of two answers is better, rather than what every pixel should be. The source of those comparisons is already inside the model. Different prompt templates produce systematically different segmentations of the same image. The authors call this prompt disagreement, and instead of treating it as noise they repurpose it as a built-in supply of preference supervision.
Adaptation then works locally. Preference queries are mined from regions where uncertainty across templates is high, the places where the prompts disagree most. The model is adapted with Region-Localized Preference Optimization, RLPO, applied to those queried regions. Alongside it sits a consistency regularization whose job is to stabilize updates outside the queried region, so that fixing one patch does not unsettle the rest of the image.
The experiments run on the MESS benchmark. Across diverse OVSS backbones the method gives consistent gains, and it does so without any pixel-level annotation at all. It also remains effective when the preferences themselves are noisy, which matters for a supervision signal that is cheap by design.
Disagreement between prompts usually reads as a weakness of a model. Here it becomes the teacher. The model's own inconsistency points at where it is unsure, a binary judgement settles that spot, and the expensive mask never has to be drawn.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A model that labels any class you name, until the images get specialized and the masks get expensive.
You have a segmentation model that labels every pixel from class names you type. It works well on everyday photos, but on medical scans, satellite images and factory parts its masks slip.
The road ahead
- 1 · The problem ~7 min A segmentation model that labels pixels from typed class names slips on specialized images, and fixing it with full masks is costly.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Different prompt wordings give different masks; ask the user which of two is right in the most disputed region, then learn from that pick.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Segment with many templates, box the most disputed region, get one binary pick, then take one RLPO-plus-consistency update.
- 6 · The experiment ~6 min You’ll guess: Preference labels from users are cheap but can be wrong. When the preferences are noisy, how does this adaptation method hold up?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.