Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

Legal Guidance in the Wild. How Users Seek Legal Help in Real-World LLM Conversations. By Amirreza Mohseni and one other, from Maastricht University. Published on the eighteenth of October. Its code repository is public but slight: two source files.

People ask conversational AI about their legal rights, their obligations, legal procedures, and personal situations that might carry legal consequences. Studying that behaviour starts with a sorting problem. A request for legal guidance has to be told apart from a conversation that merely touches on a legal theme, and from conversations that have nothing to do with law. This work supplies the tools for that sorting: a taxonomy, an annotated sample, and a reviewed evaluation set, all released publicly.

The taxonomy has several axes. One records whether legal guidance is being sought at all. Another records how the request is phrased. A third records which legal domains it concerns.

Applied to the annotated sample, it yields a first picture. Among conversations that contain legal content, seventy-one point one percent are looking for personalized or practical guidance. The most common primary type of request is advice about rights and options. And among conversations that do seek guidance, twenty-five point three percent span more than one legal topic.

The conversations are naturally occurring ones from WildChat-1M, two thousand one hundred and twenty-two of them, annotated with the taxonomy. The authors describe this as an enriched sample. On top of that sits a gold evaluation set of two hundred conversations, reviewed by humans.

The gold set is built for comparing classifiers. It supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. The annotations, the model checkpoints and the evaluation code are all available, so anyone studying legal help-seeking in conversational AI at large scale, or building models to detect it, has a starting point and a common yardstick.

One figure frames the rest. In this sample, most conversations with legal content are not discussing law in the abstract. They want personalized or practical guidance, and that is why separating guidance from mere legal talk is the first thing the taxonomy has to get right.