A Zeroth-Order Paradigm for LLM Preference Alignment

A Zeroth-Order Paradigm for LLM Preference Alignment. By Peter Chen and three others, from the University of California, Berkeley, New York University, and two other institutions. Published on the sixteenth of September. No code repository is linked to it.

Direct preference alignment methods are widely used to align large language models with human preferences, because they are efficient in computation and memory. But they face a problem known as likelihood displacement, and it motivates other ways of drawing information from preference pairs whose likelihood margins are small. The question is how to learn from those pairs by a different route.

The authors propose and analyse Comparison-based Preference Optimization, ComPO. It is a zeroth-order alignment method built on comparison oracles. Instead of directly optimizing a differentiable preference loss on those pairs, it extracts directional information from them by comparison: learning which way to move without differentiating the loss. Zeroth-order is the technical name for that: an optimizer that works from evaluations and comparisons rather than from gradients.

There are two versions. The basic offline scheme comes with a convergence guarantee, which holds under three assumptions: smoothness, sparsity of the gradient, and compatibility between the oracle and a latent objective. Online ComPO keeps the same offline comparison mechanism, and adds unlabelled generations from the policy, used for reverse-KL control relative to a reference policy. The online version, in other words, is the same comparison idea with a leash: generations from the current policy, needing no labels, keep it from drifting too far from where it started.

Following the coverage perspective on preference fine-tuning, the authors also establish a performance guarantee for a basic constrained scheme. That one holds under local coverage, and under pairwise reward accuracy within the distribution.

Experiments span Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models, and show improvements over existing direct alignment methods, including on length-controlled win rates. Diagnostics at the level of individual pairs provide evidence consistent with mitigating likelihood displacement.

The shift is in what is asked of the difficult pairs. Rather than pushing a gradient through a loss on pairs where the margin is thin, ComPO asks only which direction a comparison favours, and the guarantees spell out under what conditions that is enough. The pair-level diagnostics are described with care: consistent with mitigating the problem, not proof that it is gone.