PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Boryeong Cho, Sumyeong Ahn, Se-Young Yun
KAIST AI · KENTECH
Published August 31, 2026 · on the feed September 14, 2026 · ▲ 27 · arXiv 2608.30597
The abstract
Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.
The Brief
1:59A short podcast summary of the paper. Written and voiced by AI.
Transcript
PLC-DPO. Posterior Label Correction in Noisy and Ambiguous Preference Optimization. By Boryeong Cho and two others, from KAIST AI and KENTECH. Published on the thirty-first of August. Its code repository is public and real: fifteen source files.
Direct Preference Optimization, or DPO, makes alignment simpler by learning from pairwise comparisons: of two responses, which one was preferred. It rests on an assumption, though, that every observed preference is reliable. Real data often breaks that assumption. Some labels are reversed, some are weak, and some are simply ambiguous, and training on them as though they were clean produces harmful policy updates.
Posterior Label Correction DPO, written PLC-DPO, routes each pair's training signal into one of three cases: clean, flip, or tie. The evidence for that routing comes from the model itself. The calibrated margin between the policy and the reference is read as online evidence, and it decides which correction action a given pair receives.
The reframing is the heart of it. Noisy preference learning is usually handled by filtering out suspicious examples and training on what remains. Here the supervision is actively corrected instead. Both its direction and its strength are adjusted according to the routing, so a doubtful pair still contributes, just not in the form its original label claimed.
The comparison spans fifty-seven cells, each one a combination of dataset, model and benchmark. Across them, PLC-DPO obtains the best mean win rate against DPO: sixty point five, against fifty-five point five for the next-best method. Further checks target the routing itself: stress tests with injected noise and with ties, an analysis of human disagreement, and self-confirmation diagnostics. They show the routing staying stable, and telling flipped pairs apart from pairs that are only weakly directional.
What changes is the posture toward a bad label. Instead of something to throw away, it becomes a signal whose direction and strength can be read off the policy's own calibrated margin, and repaired while training runs.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a pile of preference labels, some of them quietly wrong.
You are aligning a language model on pairs of answers where a label says which of the two was preferred. The training loss treats every one of those labels as completely reliable.
The road ahead
- 1 · The problem ~6 min Training on pairs of answers trusts every preference label completely, but real labels are often reversed or describe two answers that are about equally good.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Treat each label as hiding a clean, flip or tie state, read the likely state from the model's own calibrated margin, and correct the pair's signal instead of dropping it.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min For each pair, compute and calibrate the policy-reference margin, turn it into clean, flip and tie weights, then mix three losses with those weights.
- 6 · The experiment ~6 min You’ll guess: Across many dataset, model and benchmark combinations, what mean win rate against plain DPO did PLC-DPO reach?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.