Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

Geometry of Values. Task Vector Composition for Ethical Preference Alignment in Language Models. By Utkarsh Agarwal and one other, from the Mohamed bin Zayed University of Artificial Intelligence. Published on the seventeenth of September. Its code repository is worked out: about forty-five source files, with a readme.

Large language models are increasingly deployed where clashing moral values have to be weighed, yet even strong models show hidden biases and brittle instruction-following across languages. The work makes two kinds of claim. One is behavioral: models carry systematic preferences and biases on value dilemmas. The other is geometric: the direction for a specific value preference can be isolated inside a model and used, through task arithmetic, to produce a model with the opposite stance.

The authors build a dataset of twelve thousand two-option dilemmas, covering three pairwise value conflicts: honesty against justice, justice against autonomy, and autonomy against honesty. It is translated into Hindi, Arabic, Spanish, and Chinese, to probe behavior across languages. Benchmarking GPT-5-mini shows that when no policy is given, it consistently favors honesty over autonomy, in all five languages. Two Llama-3.2 models show a strong first-option bias. Both plain fine-tuning and fine-tuning with Direct Preference Optimization remove that bias, raising accuracy above ninety-eight percent.

High accuracy after fine-tuning leaves an open question: has the model learned a value, or only correlations in the dataset? To separate the two, the authors run a task-vector transfer experiment. They compute the task vector for one direction of value preference, then orthogonalize it against the general instruction-following vector. What remains isolates the specific value preference, and task arithmetic with that direction yields a model holding the opposite stance. Put simply, the part of the change that any instruction tuning would produce is removed, and what is left is attributed to the value itself. If that remainder can be flipped to produce the opposite preference, it is carrying the value rather than a quirk of the data.

The dilemmas have two options each, and the value space is three values, in three pairs. The cross-lingual preference result is reported for one model, and the bias result for two small open models. The task-vector finding is the outcome of that one experiment design, removing the instruction-following component before transferring the value direction.

The title's geometry is meant literally. Once the part a value direction shares with plain instruction following is projected away, what is left behaves like a direction in the model, one that can be pointed the other way.