MMDM: Measuring Massive Multitask Decision Making

MMDM. Measuring Massive Multitask Decision Making. By Junhao Shen and five others, from the Chinese University of Hong Kong, the Hong Kong University of Science and Technology, and two other institutions. Published on the tenth of November.

A System-1 Decision Model does not write out its reasoning. Given a state, it makes a structured probabilistic decision over options specified by the query, in a single forward pass, without generating reasoning trajectories or text. Recent evaluations of such models cover diverse tasks, but they often rely on existing benchmarks, and prior exposure to that test material can obscure how well a model generalizes. Real-world decisions also tend to combine long contexts, large option spaces and complex logical dependencies, and existing benchmarks do not capture those challenges together. A benchmark can test long inputs, or many options, or tangled logic, but a real decision often brings all three at once, and a model that copes with each one separately may still fail when they arrive together.

MMDM, short for Massive Multitask Decision Making, measures decisions of three kinds: single-select, multi-select, and soft-label, all posed in fresh contexts. Its scenarios are meant to be realistic. They include agent state tracking, tool and skill selection, preference judgments, and rubric scoring. The three formats ask different things of a model. A single-select question wants the one right option. A multi-select question wants every right option and none of the wrong ones. Soft-label decisions round out the set, fitting models whose output is a probabilistic decision in the first place.

The benchmark holds five thousand four hundred and forty-eight high-quality questions across sixteen task categories. Some tasks are newly constructed. Others are source-grounded recompositions and rewrites, which is how the material is kept fresh rather than lifted from earlier test sets. Option spaces range from two choices up to one thousand and twenty-four, and inputs extend to sixty-four thousand eight hundred and seventeen tokens.

Twenty-eight decision models are evaluated. The broad pattern is that many do well on common knowledge, and struggle with interdependent options, constrained decisions, and multi-select tasks. Knowing a fact is one skill. Weighing options that bear on each other, or picking under a constraint, is another, and the results suggest the second lags well behind the first.

One model shows the slope clearly. Perplexity Decider-V1.2 reaches seventy-eight point seven five percent Hard Single-select Accuracy on world knowledge. On option logic it drops to fifty-three point nine five percent, and on constraint planning to forty-seven point nine five percent. Multi-select is harder still. The best Hard Multi-select Accuracy across all the models is thirty-six point eight one percent. That number is a ceiling, not an average: no model among the twenty-eight recovers the full set of correct options on even two hard multi-select questions in five.

Read as a list of priorities, the results point at three things decision models do not yet do well: reason jointly over options that depend on each other, select while respecting constraints, and recover every correct answer when more than one is right. MMDM is offered as the testbed for measuring progress on exactly those.