Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

Bridging Exploration and Persona. Information-Directed Dueling for Test-Time Agent Personalization. By Jie Mao and six others, from the Chinese University of Hong Kong, Shenzhen, the University of Copenhagen, and two other institutions. Published on the seventeenth of October. Its code repository is public and real: eighteen source files.

For an agent built on a language model to personalise itself at test time, it has to adapt to user preferences that depend on the query, and it has to do so from only a little interaction. The setting studied here is persona selection. The online signal is a binary comparison: of two candidate responses, which one is preferred. The persona space to choose from is large and discrete.

ID-TAP is a scalable neural dueling-bandit framework for that setting, inspired by information-directed sampling. It pairs a neural utility model conditioned on the query with a lightweight Bradley-Terry explorer. Evidence from comparisons is carried across related queries through warm starts weighted by similarity.

The two parts divide the work. The neural model shares statistical strength across queries while still keeping scores specific to each query. The efficient Bradley-Terry updates are used to score hypothetical challengers, the candidates that might be put up for the next comparison.

ID-TAP is tested on seven user-conditioned generation benchmarks, under both reference-based and model-based preference simulators. Within a fixed budget of ten comparison rounds, it achieves the strongest aggregate performance among the baselines evaluated.

The authors take this as support for choosing challengers by information as an effective design for adapting personas at test time. With only ten rounds to spend, which comparison to ask for next is most of the problem.

Bridging Exploration and Persona. Information-Directed Dueling for Test-Time Agent Personalization. By Jie Mao and six others, from the Chinese University of Hong Kong, Shenzhen, the University of Copenhagen, and two other institutions. Published on the seventeenth of October. This is the long version: the whole paper rather than the abstract.

What you want from an assistant shifts from one request to the next. A creative collaborator suits brainstorming; a rigorous critic suits editing. The paper treats that problem as a choice among persona prompts: short descriptions of the style and stance the agent should adopt. The catch is the feedback. At test time the only signal is a binary comparison: the agent shows two responses, and the user picks one. With four hundred candidate personas per user and a budget of ten comparisons per query, most personas will never be tried. Every comparison has to be chosen so that it teaches the agent something it does not already know.

Three ideas carry the rest. The first is the persona pool. Offline, a language model writes a persona summary from the user's history, then rephrases it again and again with bounded variation, resetting to the original history every fifteen steps so the descriptions do not drift. The result is a dense pool: the mean pairwise cosine similarity sits between about point six six and point seven two across the benchmarks, and the pools are only weakly clustered. Close descriptions, the authors note, can still produce meaningfully different responses.

The second is the dueling bandit. Each round picks two arms, a champion and a challenger, and observes only which one won. The third is Information-Directed Sampling, which weighs an action's estimated utility gap, squared, against the information it is expected to yield, and picks the smallest ratio.

ID-TAP splits the work between two systems. System one is a small neural network that maps a query embedding and a persona embedding to a utility score, and persists across a user's queries. Each round, its top persona becomes the champion, and the gap between the champion's score and any other persona's is that persona's estimated cost of being shown.

System two is a lightweight Bradley-Terry model kept separately for each query. It holds only win counts and comparison counts between pairs, smoothed toward a coin flip, and fits a strength for every persona with a fast minorization-maximization solver. Those strengths, passed through a softmax, become a distribution over which persona is best, and its entropy measures how unsure the explorer still is.

The challenger is chosen by imagining the duel. For each candidate, the explorer adds a hypothetical win for the champion, refits, and records the new entropy; then it does the same for a hypothetical loss. Weighting the two by the predicted chance of each outcome gives the expected entropy afterwards, and the drop from the current entropy is the information proxy. The challenger is the persona that minimises its squared utility gap divided by that proxy. Two refits per candidate per round is why the solver must be cheap; the neural network updates only once per real comparison.

One more piece links queries. A new query inherits Bradley-Terry counts from earlier queries, each weighted by a sigmoid of its embedding similarity, and the weights are capped so the inherited evidence never exceeds one query's worth. After ten rounds, the neural network's top persona writes the final answer.

Evaluation spans seven benchmarks, from LaMP headlines and titles to LongLaMP long-form writing, PrefEval and UltraChat, with DeepSeek-V3.2 generating. Real users were not involved: a proxy user supplies the binary choice, either as a Bradley-Terry draw on ROUGE-L against a held-out reference, or through an LLM judge shown that reference. Each configuration is run three times per user.

ID-TAP is best in thirteen of the fourteen dataset and metric cells. Its average normalised ROUGE-L is point nine one nine against point eight five two for POHF, and its win rate against a naive response is eighty-three point six percent against seventy-nine point three for PDO Sampling. The one loss is the LongLaMP abstract task, where PersonaAgent wins on win rate. On the averaged learning curve, it overtakes the baselines after about four rounds.

The comparison that tests the thesis most directly is the ablation that keeps everything except the challenger rule, replacing it with a uniform random draw. Full ID-TAP wins by point zero zero five, point zero two and point zero one one on three tasks. The authors call that a modest contribution, and it is. The two history mechanisms add similar small amounts on the LongLaMP tasks and nothing on PrefEval.

There is no limitations section. The conclusion names two gaps as future work: testing the interface with real end users, and a finite-sample analysis of the combined neural and Bradley-Terry procedure. Both matter. Every preference comes from a proxy user that can see the reference answer, so learning from lightweight human feedback is untested on humans.

The ablation covers three datasets without standard errors, and its margins sit in the second or third decimal place. Several main win-rate gaps fall within one standard error. Robustness checks with GPT-4o judges and BERTScore cover two to four datasets each. On cost, the algorithm takes up to about point six of a second per round, between seven and twenty-two percent of total latency, measured with a pool of one hundred personas rather than four hundred.

The idea to keep is the division of labour. One model learns, slowly and globally, which persona is good. A second, deliberately crude model answers a different question: which duel would settle the most uncertainty right now. The crude model can afford to imagine both outcomes for every candidate because it is only counting wins. In a pool of near-identical personas, that imagined duel is what keeps the agent from asking the same question twice.