Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG. By Parv Khatri and three others, working independently. Published on the twenty-seventh of October.

When the sources retrieved for a query disagree, the right response depends on why. A question with several valid answers, a contested topic, stale information and a false source each call for different behaviour. Earlier work by Cattan and colleagues showed that telling a generator the conflict type improves its responses by about twenty-four points, but that predicting the type is hard even for frontier LLMs. This study asks whether the prediction needs an LLM at all. Typed decision models return probabilities over fixed options in a single forward pass instead of generating text, and the answer is that they can match the model used in that earlier work, faster and far more cheaply, though not the newest one.

The setup holds everything else fixed. Using the CONFLICTS benchmark and the original paper's prompt word for word, the authors evaluate all eleven decision models served by one commercial inference gateway, drawn from seven providers, against Gemini models on identical inputs. Asked directly, the decision models reach between fifty and sixty-seven percent accuracy. Ten of the eleven match or exceed Gemini two point five Flash, the original paper's model, which scores sixty-one point three percent. They do so at four to thirty-nine times lower billed cost, with two of them free, and at two point six to six times lower latency.

The current Gemini three point eight Flash, at seventy percent, is significantly more accurate than eight of them. A cascade narrows that gap. Send each decision model's least confident quarter of queries to the Gemini model, and for eight of the eleven the result is statistically tied with it, at twenty-five to thirty-seven percent of its cost. End to end, answers generated with the types from ten of the eleven decision models follow the expected behaviour as often as answers generated with the LLM's types.

There are limits, and the study charts them. Breaking the taxonomy into yes or no questions lowers accuracy for every model, and a repair fitted on one model does not transfer. Pre-specified diagnostics show that dates play no measurable role, while sensitivity to the order of sources varies widely between models. Decision models also detect a planted false source less reliably than the LLM when read as hard labels, though their probabilities still separate planted sources from clean ones. All of this rests on one benchmark, one prompt, and one gateway's catalogue of models.

The title's question gets a qualified no. For knowing why sources conflict, a classifier that outputs probabilities can stand in for a generating model on most of the work, and its confidence is useful in its own right: it picks out which queries deserve escalation. The expensive model becomes the exception rather than the default.