GraphDecisionBench: Benchmarking System One Models on Graph Tasks

GraphDecisionBench. Benchmarking System One Models on Graph Tasks. By Xianliang Yang and two others, from Microsoft and the Beijing University of Technology. Published on the ninth of October. Its code repository is worked out: about eighty source files, with a readme.

Large language models are being explored more and more for understanding graphs and making decisions on them. Alongside them sits a different kind of model, which the authors call System One: models such as Jev, which select directly from the options they are given. How well that kind of model handles graph tasks has remained unclear.

The benchmark, called GraphDecide in the abstract, is built to diagnose graph decision performance, and it is model-independent. It does not reduce that performance to one number. It asks separate questions: about structure, about the effect of the input format, and about the quality of what gets built.

Three ingredients carry those questions. Structural task profiles. Matched contrasts between graph input and text input. And controls that use heuristic proposals. Task contracts, candidate interfaces and scoring rules are fixed, so native selectors and language-model adapters can be compared on the same terms. The evaluation covers Jev and related choice-based models next to language-model baselines, across fourteen model-interface configurations.

Jev's results are used to illustrate the distinctions the benchmark draws. Recognising adjacency accurately does not guarantee broader structural correctness. Giving graph and text input together does not consistently improve prediction. And producing a feasible construction does not establish that the solution is of high quality.

Each of those is a pair of things a single score would merge, and the benchmark keeps them apart. Code and aggregate results are released.