WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

WearableQA. A Benchmark for Health Reasoning over Real-World Wearable Data. By Ji Soo Lee and seven others, from Meta, KAIST, and one other institution. Published on the fourth of September. Its code repository is public but slight: one source file.

Wearable sensors now monitor physiological and behavioural signals continuously, day after day. Existing benchmarks, though, rarely test whether an AI system can reason over one real user's longitudinal wearable record: the long, messy history of a particular person, rather than a tidy snapshot. A system that will one day answer health questions from a person's own wearable data has to work with measurements that span months, carry the quirks of real devices, and differ from one person to the next. Without a benchmark built from that kind of record, there is no way to tell how well current models cope with it.

WearableQA is built to test exactly that. Its question types are organised along two axes that complement each other. The first axis separates data reasoning from health reasoning: computing over longitudinal measurements on one side, interpreting them physiologically on the other. The second axis separates single-signal reasoning, about one measurement stream at a time, from cross-signal reasoning, which has to integrate several. Sixteen question types are spread across those axes, so a model's score can be read by the kind of reasoning it demands.

The benchmark holds four thousand and eighty-four multiple-choice questions, each with ten options. They are built from the data of two hundred real users: their wearable time series, blood biomarkers and demographics, with up to five hundred days of daily measurements per user. Because the data are real, the benchmark keeps authentic wearable distributions, including device noise and the variability between individuals.

To construct reliable questions at scale, the authors use a dual-grounding framework. It combines physiological findings grounded in the literature with population-level physiological patterns that have been statistically validated. That pairing is what lets the questions capture relationships that actually show up in real-world wearable data.

Fourteen language models were evaluated, proprietary and open-source. The benchmark separates them clearly: performance ranges from nineteen point six percent to seventy-two point nine percent, against a chance baseline of ten percent. It is also far from solved. Most of the models score below sixty percent, which leaves a wide gap between today's models and a benchmark that has been mastered.

The value here lies in realism and in diagnosis. Real users, real noise and long records make the task resemble what a health assistant would actually face, and the two axes make a low score informative, pointing to whether a model struggles with the arithmetic of a time series or with what the numbers mean for the body.