OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

OpenTumorBoard. A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories. By Anqi Li and eleven others, from Rice University, the University of Illinois at Urbana-Champaign, and four other institutions. Published on the twenty-sixth of September. Its code repository is worked out: about sixty source files, with a readme and a licence.

Cancer care often runs through a multidisciplinary tumor board, where specialists discuss a patient together. Into that one conversation they bring multimodal clinical observations and the patient's longitudinal history, and out of it comes a shared plan. The path from the first remark to that plan is what the authors call a discussion trajectory. Benchmarks, they note, rarely capture these real-world trajectories. A model can be tested on medical questions in isolation without ever being asked to take part in the back-and-forth by which a board actually reaches its view.

OpenTumorBoard is built from that back-and-forth. It holds six hundred and eleven patient cases and nineteen thousand one hundred and fifty-seven discussion turns, spread across ten specialist roles. All of it was transcribed from twelve thousand five hundred and thirty-four minutes of tumor board recordings that are publicly available on YouTube. The curation is automated, handled by a pipeline the authors also plan to release.

The benchmark poses two settings, at two different scales. The first, called SPECIALIST TURN, is narrow: a model answers a clinically significant question that was posed at a particular moment during a real discussion, and its answer is set against the specialist's. The second, BOARD SIMULATION, is broad. The model generates an entire back-and-forth discussion itself and has to reach a consensus. That consensus covers therapy recommendations, surgical plans, next actions, and clinical trial matching.

Because the material is mined automatically from recordings, its quality is checked by people. Three M.D. experts reviewed a subset of the benchmark. They found high information coverage and factuality in the patient cases, and strong fidelity in the consensus conclusions that had been extracted.

Fourteen models were evaluated, a mix of general-purpose frontier LLMs and medical ones, and the limitations are substantial. On clinical equivalence to the specialists' answers, the best models score three point four three out of five. On alignment with the conclusions the boards actually recorded, the best reach two point seven eight out of five. Supervised finetuning and reinforcement learning both improve performance on a held-out test set, which suggests that real discussion trajectories can support adapting a model.

Neither score sits near the top of its scale, even for the strongest models. And the training result means the recordings serve as more than a test: they can also teach. The authors offer OpenTumorBoard for both the development and the evaluation of LLMs aimed at multidisciplinary, personalized cancer decision-making.