MA-Bench: Measuring Operational Planning in Language Models

MA-Bench. Measuring Operational Planning in Language Models. By Rodney Lafuente-Mercado and three others, from Scale AI and Data Machines. Published on the thirteenth of October. No code repository is linked to it, though it does publish its weights.

The question here is how well a language model can take a large body of documents, incomplete and partly contradictory, and turn it into a plan, when there is no single correct answer to check against. That kind of task resists ordinary scoring, and it is the kind of task this benchmark is built around.

Mission Analysis Bench, or MA-Bench, draws on mission analysis: the first step of military operational planning. In it, a staff reads the full situation and states the problem before anyone proposes a solution. The benchmark grades the determinations a model makes, not the quality of its prose.

There are ten expert-built scenarios. Each pairs a synthetic packet of ten to twelve documents with requirements written by experts, one thousand two hundred fifty-seven pass or fail items in all. Every requirement records its source, its question type, and its importance level.

Grading is done by a reasoning-model judge, which scores each brief against the requirements without seeing the documents. Two checks support it. Added to a panel of four credentialed planners, the judge behaves like a fifth: the panel's agreement barely moves, a Fleiss kappa of zero point four two becoming zero point four one. And a judge from a second provider reproduces the same model ranking, with a Spearman correlation of zero point nine four.

Twenty frontier and open-weights models are tested. The best gets sixty-two percent of the rubric items right. Performance splits by kind of item. Models are strongest on facts stated in the documents, at fifty-eight percent, and weakest on judgments that have to be inferred, at thirty-three percent. Six failures in every ten are omissions rather than errors: the model left something out more often than it got something wrong.

The headline is a ceiling well short of full marks, and the shape of the misses matters as much. Models do better at retrieving what a document says than at concluding what it implies, and they fail mostly by silence. The authors release the packets, rubrics, prompts, scorer, and every graded brief with its item-level verdicts.