PACE: Towards Surfacing Hidden Conflicts in User Requests

PACE. Towards Surfacing Hidden Conflicts in User Requests. By Yoojin Kim and two others, from POSTECH. Published on the third of September. Its code repository is public and real: seventeen source files.

A personalized assistant should do more than carry out what it is asked. It should assess whether the request is appropriate given the user's current circumstances, and refuse when it is not. Prior work has concentrated on executing requests accurately and largely skipped that judgment. What now exists is a dataset built to evaluate exactly it: whether a model can identify latent constraints that make a perfectly reasonable-sounding request inappropriate for this particular person right now.

The dataset pairs user requests grounded in well-defined personas with egocentric knowledge-base facts, meaning facts about the user's own life and events. A model has to integrate that contextual evidence to decide whether a request conflicts with it. The design decision that gives the benchmark its difficulty is that the constraining factors are implicit rather than stated. They have to be retrieved from a knowledge base, and existing work on conflict and safety detection has generally relied on factors being handed over explicitly.

That implicit retrieval setting is deliberately adversarial to the obvious approach. Because the conflict-inducing knowledge is not lexically connected to the request, a model cannot find it by association, which is what makes identifying the relevant user-specific facts hard for existing systems. The authors also build a multi-agent framework as a response, with specialized agents coordinating across query reformulation, multi-hop graph traversal, and conflict-aware filtering, in order to retrieve the evidence that actually decides the question.

Experiments evaluate two things separately, evidence retrieval quality and conflict decision accuracy, which matters because a system can reach the right verdict from the wrong evidence. The proposed framework consistently outperforms existing approaches on both. Separating the two also tells you where a system fails when it does: retrieving the wrong evidence and reasoning badly from the right evidence are different problems needing different fixes, and one combined accuracy number would hide which of them you have.

The idea underneath is that refusal has been studied as a safety property of requests in the abstract, when a great deal of it is really a personalization problem. Whether a request is appropriate can depend entirely on a fact about the person asking that nobody has said out loud.

PACE. Towards Surfacing Hidden Conflicts in User Requests. By Yoojin Kim and two others, from POSTECH. Published on the third of September. This is the long version: the whole paper rather than the abstract.

The paper's worked example is the clearest statement of the problem, so start there. A user asks an assistant to book a table at a restaurant on Friday from seven to eight, for two people. Nothing about that request is suspicious. It contains no risk, no harmful content, nothing an input filter would catch. It is also, given this particular user's circumstances, the wrong thing to do. That gap is the whole subject: an assistant that only inspects what it was asked cannot see it, however carefully it inspects.

Why becomes clear only from the user's personal knowledge base. He already has a Friday dinner plan with a companion. That companion is vegan. The restaurant's fixed menu this week is steak, bone broth and cheese-based dishes, and it does not offer ingredient substitutions for the fixed menu. Four separate facts, none mentioned in the request, which together make a perfectly ordinary booking inappropriate. And they sit in a knowledge base alongside plenty of plausible distractors: that he enjoys steak, that he prefers quiet restaurants, that this restaurant is indeed quiet between seven and eight, that it served a seasonal vegetable dish last year, that he listens to jazz on Friday evenings.

Existing benchmarks for safety and risk-aware reasoning focus on detecting risk or harmful content explicitly present in the input. That is useful for single-query risk recognition and it fails to capture this setting entirely, where making an appropriate decision means identifying and integrating the handful of facts that truly matter out of a large personal knowledge base full of plausible irrelevancies. So PACE pairs user requests grounded in well-defined personas with egocentric knowledge-base facts, and asks whether a model can find the latent constraints that render a reasonable-sounding request inappropriate.

The authors are equally clear about the failure in the other direction, and this is what keeps the work from being a refusal benchmark. An assistant that over-refuses plausible requests forces users into unnecessary follow-up interactions, degrading both usefulness and efficiency. The stated target is a situated decision: neither blindly executing nor conservatively refusing, but grounding the judgment in evidence about the user's actual circumstances. Their proposed system is a multi-agent framework whose specialized agents coordinate across query reformulation, multi-hop graph traversal, and conflict-aware filtering. Each of those addresses a specific part of the difficulty: reformulation because the request's own words will not find the evidence, traversal because the decisive facts connect through intermediate entities, and filtering because plenty of retrievable facts argue the wrong way.

On both measures the proposed framework consistently outperforms existing approaches, and the code and dataset are released. The structure of that result is what to take from it rather than the margin. A gain on retrieval says the decisive facts can be found when the search is reformulated and traversed rather than matched; a gain on decision accuracy says that having found them, integrating several into a single verdict is a separable skill. A system strong on the first and weak on the second would be one that surfaces the vegan companion and the fixed menu and still books the table, which is a failure mode no combined score would have named.

The authors are direct about scope. The benchmark focuses on feasibility judgment rather than full task execution, which isolates the problem of detecting latent conflict signals but does not evaluate whether an assistant can then complete the downstream task, whether that is recommendation, scheduling or planning. Knowing that a booking should be refused is not the same as handling the situation well. They point at richer persona worlds, actionable entities and dynamic task environments as what a fuller benchmark would need. And their own method is evaluated without task-specific training, which they present as evidence of generality while noting that specialized trained agents for reformulation, selection, traversal and calibration could do better.

The reframing is that a large part of what gets discussed as safety is really personalization. Whether a request is appropriate can turn entirely on a fact about the person asking that nobody has said out loud, and no amount of scrutiny applied to the request itself will find it.