Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters. By Constantinos Karouzos and one other, from the University of Sheffield. Published on the sixth of November.

A deployed language model is often asked to balance objectives that compete, and the right balance shifts as user preferences change. Standard post-training does not accommodate that well. It typically produces a single policy tuned for one fixed weighting of the objectives, so a different trade-off means a different training run.

LambdaMix is a parameter-efficient method that learns not one policy but a whole family of them, conditioned on a preference, using group-relative online reinforcement learning. The family is built from low-rank adapters, one per objective, trained jointly on top of a frozen base model.

A single preference vector does two jobs at once. It selects the mixture of adapters that generates the responses, and it weights the rewards used to update those adapters, rewards that are normalized separately for each objective. A single training run is enough to support changing preferences afterwards: at inference the preference can be changed without any further optimization.

The experiments use four models on two tasks: helpfulness against harmlessness, and solving math problems while following instructions. The two largest models are also tested on a third task, rewriting comments so the wording is preserved while clues about personal attributes are reduced.

On the first two tasks, LambdaMix reaches higher hypervolume than every conditioned baseline and every policy merging baseline evaluated, and it does so across all four models. It also uses fewer training completions, and stores fewer adapter parameters, than a set of GRPO specialist policies would need. A further comparison holds architecture and initialization fixed. There, varying the adapter mixture and the objective weights together gives wider score ranges and higher coverage than varying either one alone.

The design rests on a single coupling. The preference that chooses how the adapters are blended is the same preference that decides how the rewards are weighted, and the last experiment suggests that coupling is doing real work: moving both together beats moving either by itself. The result is one trained artifact that can be steered at inference instead of a shelf of specialists.