MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching

MoveBench. A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching. By Yitong Li and four others, from Keep. Published on the twenty-sixth of October.

Large language models have advanced quickly, but putting them to work in sports and health remains high-risk. The reason the authors give is simple: there are no evaluation benchmarks built for the domain. General-purpose benchmarks, MMLU and HELM among them, mostly assess broad knowledge and reasoning. They miss requirements that matter here, such as exercise safety, physiological appropriateness, personalization, and whether the coaching is actually actionable. In fitness and health applications that gap is more than academic. A wrong or unsafe recommendation can lead to physical harm.

MoveBench is presented as the first comprehensive benchmarking framework designed specifically for evaluating LLMs in sports and health coaching. What it measures is coaching quality as the domain defines it, with structured rubrics that emphasize three things: safety compliance, scientific validity, and sound training logic. That moves the question away from whether a model knows facts about exercise and toward whether its advice is safe, grounded in science, and coherent as a training plan. Those are the requirements the authors say general benchmarks fail to capture, and here they become the criteria an answer is graded against.

The benchmark mixes heterogeneous task formats. There are multiple-choice questions, and there is open-ended question answering. Mixing the two lets the benchmark look both at picking a right answer from options and at producing coaching in free text. Both formats are grounded in real-world user scenarios, the kinds of situations a person seeking coaching might actually bring, and each is paired with a structured rubric.

The scale is concrete. The benchmark holds eight hundred and nine questions. Across them sit five thousand and twelve rubric-level criteria, so a single answer is judged against many specific checks rather than one overall grade.

The authors run a large-scale evaluation of fifteen diverse LLMs. Performance varies substantially. The variation is sharpest in three places: awareness of safety, the design of structured training programs, and the ability to personalize. Taken together, the results point to what the authors call a critical gap between knowing and doing. A model can hold the knowledge and still fall short when asked to turn it into a coaching plan.

MoveBench aims to be a foundational evaluation standard for sports and health applications of language models. Its framework is meant to be reproducible, so that work toward safe, reliable and context-aware coaching systems has a common yardstick for the qualities general benchmarks leave out.

MoveBench. A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching. By Yitong Li and four others, from Keep. Published on the twenty-sixth of October. This is the long version: the whole paper rather than the abstract.

Ask a language model for a training plan and a wrong answer is not just wrong. It can hurt someone. A plan that ignores a sore knee, or a crash diet that sounds plausible, is a physical risk, not an informational one. The benchmarks people use to judge these models were not built for that. MMLU and HELM measure broad knowledge and reasoning. Medical sets like MedQA and HealthBench treat the model as an informational assistant or a clinical reasoner. The authors, from the fitness platform Keep in Beijing, argue that coaching is a different job: the output has to become an executable plan, fitted to one person's body, injuries, time and space, and it has to be safe. Their diagnosis is a gap between knowing and doing, and MoveBench is built to measure it.

Two axes organise the benchmark, and the paper insists they not be confused. The first is capability domain. Plan generation asks for a structured workout under real constraints. Data analysis hands the model actual activity logs, with pace, heart rate, cadence and perceived exertion, and asks for physiologically grounded insight. Knowledge question answering covers the everyday questions users bring, often short, ambiguous or built on a false premise. The second axis is question format: single-select, multi-select, and open-ended. The multiple-choice items are scored by exact match, with no partial credit. Open-ended items are scored against a rubric written for that item. In total there are eight hundred and nine questions and five thousand and twelve rubric criteria. Four hundred and ninety-nine of the questions are open-ended.

Questions are generated, then filtered. Gemini three point zero pro writes candidate items at a moderate temperature. The user in each scenario is not invented freely: age, gender, body mass index and training habits are sampled from weighted distributions drawn from real platform data. A language model assessor then checks each item for clarity, a correct and unique answer, and consistency with its rubric. Anything rated moderate or high risk is rejected automatically, and the rest go to domain experts who approve, revise or reject.

Scoring is where the design gets careful. Each open-ended answer goes to a committee of three judges: Gemini three point one Pro, Claude four point six Sonnet, and GPT five point two. Each scores every criterion with a rationale. Their scores are combined by fixed weights, thirty-five percent each for Gemini and Claude and thirty percent for GPT. Safety is handled separately. Each judge can cast a redline vote for medical overreach, unsafe guidance, or an invented physiological concept. If at least two of the three judges vote redline, the whole item scores zero. One vote alone changes nothing. The penalty value of minus ten that appears in the rubrics is only the encoding of that vote, never a deduction. The worked example in the appendix shows the texture: a recovery walk for a woman with mild joint pain, where the plan must stay within six to ten minutes, keep heart rate near a hundred beats per minute, and fit a small outdoor space. Scores are then pooled within each format, and the overall figure combines formats by how many items each had, so the open-ended questions carry most of the weight.

Fifteen models were evaluated. Claude four point six Opus scores highest overall at zero point nine four three, and the lowest, GPT six, scores zero point eight seven five. The headline spread is narrow. The more telling result is by format. Every model scored a perfect one on the single-select questions it answered: a ceiling, as the authors say, not mastery. On open-ended questions the range widens to between zero point eight one two and zero point nine one seven. The sharpest case for the thesis is Gemini four Argon. It has the best multi-select accuracy of all fifteen, at zero point nine eight, and the worst open-ended score. Picking the right options is not the same as writing a safe, fitted plan.

The paper then tests its own scorer, against thirty open-ended questions and three hundred criteria that three experts scored and a senior expert adjudicated. Each expert agreed with that reference on eighty to eighty-six percent of criteria. The judge models agreed on sixty-two to seventy-six percent. Run three times, the judges' scores also drifted: GPT five point two moved across a range of zero point two one three, Claude Sonnet the least.

The paper has a limitations discussion, and it is frank. It names the judge alignment gap, sixty-two to seventy-six percent against eighty to eighty-six, and says judges diverge most on borderline safety and partial personalization, which is exactly where the benchmark claims its value. It names the saturation of the objective questions, which leaves little power to separate models. It reports bookkeeping gaps in its own tables, such as twenty Claude criteria left unclassified, and declines to guess the cause. It notes that the chosen balanced weights are not the lowest error configuration, which put sixty percent on one judge. Beyond what is stated: GPT five point two is both a judge and one of the fifteen models judged, and the paper does not examine that overlap. The validation responses all came from one model, Qwen three point five twenty-seven B. And the conclusion says it outright: this is a basis for evaluation, not a certification of safe coaching.

The idea to carry away is that in a safety-critical domain the format of the question decides what you learn. Multiple choice says these models know their fitness facts, and they all do. Open-ended coaching, scored criterion by criterion with a veto that needs two judges to agree, is where they come apart. The benchmark's own reliability numbers are the reminder that such a veto is only as good as the judges casting it.