Collaborative Personalized Preference Alignment for LLMs under Data Deficiency

Collaborative Personalized Preference Alignment for LLMs under Data Deficiency. By Liyan Yang and two others, from the City University of Hong Kong, the University of Washington, and one other institution. Published on the fifth of October. No code repository is linked to it.

Real users want different things from an LLM's responses, and their preferences span several objectives at once. A lightweight aligner can tailor responses to one person, but each user supplies only scarce feedback, which makes personalized training difficult. Learning a shared initialization across users is one way to make few-shot adaptation possible. The trouble is that heterogeneous preferences and competing objectives produce gradient conflicts, both between users and within a single user, and those conflicts get in the way of learning a good initialization.

The question the authors pose is how to learn aligner initializations collaboratively, so that they support few-shot adaptation to diverse preferences. Their answer is Approximate Pareto Optimality, or APO. Its guiding principle is to combine information only where it is compatible, and to aim each starting point at what the users sharing it actually want.

APO starts by grouping users whose updates are compatible, so their information can be combined with less interference. Within each group, it combines gradient descent with controlled ascent. That coordinates the competing objectives and moves toward points on the Pareto front specific to each preference. The result is an initialization that sits close to the optima of the users in the group. APO then refines it iteratively, using updates from few-shot local adaptation, so that it becomes a more effective starting point for personalization.

On the theory side, the authors establish conditional suboptimality bounds for a collaborative update with one local step. They also characterize how error in the initialization carries through into the stochastic adaptation that follows it.

Experiments on Fed-ChatbotPA and UltraFeedback show consistent improvements over existing methods, using only twenty local examples.

The paper locates the difficulty in conflict as much as in scarcity. The little signal that exists pulls in different directions, across users and across objectives, and a single shared starting point inherits all of that tension. Grouping first, then coordinating inside each group, is how APO keeps the conflict from erasing what each user contributes, so that twenty examples are enough to finish the job.