Clinician use of language models diverges from how the models are evaluated

Clinician use of language models diverges from how the models are evaluated. By Krithik Vishwanath and twenty-one others, from NYU Langone Health, New York University, and six other institutions. Published on the eighth of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

Large language model assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores. Most of those benchmarks are derived from examination questions or curated cases. A benchmark predicts performance in deployment only to the extent that its items resemble real use, yet whether benchmarks reflect the work these systems actually receive has rarely been measured. This study measures it, and the claim is stated in the title. Clinician use of language models diverges from how the models are evaluated.

The usage side is real traffic: one hundred and twenty-seven thousand eight hundred and thirty-three queries, sent by six thousand three hundred and forty-two physicians, advanced practice providers and nurses. They work in thirty-five specialties, and they used an institutional assistant during an eight-month roll-out. Each query is characterized with RCQ-Map, a clinician-validated framework grounded in taxonomies of clinical questions and of LLM evaluation. It records the query's task, its intent, its answerability, the information missing from it, and its potential harm.

Here is what clinicians actually asked for. Documentation and administration made up thirty-six point two percent of use, and knowledge retrieval twenty-eight point nine percent, together nearly two-thirds. Diagnosis was three point seven percent. More than a third of queries could not be answered well as posed.

The same lens was then turned on fifty-eight public benchmarks, drawn from major evaluation suites and from frontier model reports, and assembled into the Clinical AI Benchmark Atlas. The median benchmark contained no documentation requests at all, and shared thirty-one percent of the task mix of real use. That is less overlap than an even spread across task categories would give. Benchmarks in suites designed to resemble clinical practice were, individually, no closer to real use than those used in frontier model reports.

The boundaries matter. The picture of real use comes from one institutional assistant over eight months, so it reflects that institution's clinicians and specialties. The benchmark side covers the fifty-eight benchmarks gathered in the Atlas. And the comparison is one of resemblance, between what benchmark items ask and what clinicians ask, measured through the categories RCQ-Map records. From that mismatch the authors conclude that benchmark scores say little about how clinical AI performs on most of the work it is actually given.

Their recommendation follows directly: evaluation should be matched to real clinical use. The sharpest detail is an inversion of emphasis. Diagnosis, the kind of task examination questions are built around, was under four percent of what clinicians asked, while documentation, more than a third of real use, does not appear at all in the median benchmark.