Measuring Collapse and Correction in Homogeneous-Panel LLM Debate

Measuring Collapse and Correction in Homogeneous-Panel LLM Debate. By Xin Li and two others, from Nanyang Technological University. Published on the twenty-eighth of September. Its code repository is worked out: about seventy source files, with a readme and a licence.

Debate among several large language model agents is usually judged by whether the final answers improve. But movement is not the same as improvement. One discussion can rescue a majority that started out wrong, and the same kind of discussion can destroy a majority that started out right. Standard final-accuracy evaluations fold these opposing mechanisms into one number and hide both.

The authors introduce an auditable protocol for homogeneous debate on multiple-choice questions. It records each run as a transition ledger over four entries: collapse, correction, onset, and signed intervention utility. Every debate becomes an account of what moved, in which direction, and when, and any intervention can be charged against both sides of that account.

The protocol is applied to six thousand nine hundred and twenty-five debates on MMLU-Pro. Replay experiments then test interventions against the ledger. A compact screen of eight probes, run before the debate starts, is examined as an early warning, and traces at the level of individual rounds show where in a debate things change.

The ledger identifies two hundred and fifty-three collapses, together with a parallel correction ledger that changes how interventions should be judged. Replays expose the central tradeoff. A probe-gated freeze, tested leave-one-model-out, prevents twenty-nine collapses but loses one hundred and eight corrections under equal weights. Counting only collapses prevented, it looks like a win. Counting both, it is the wrong policy.

The eight-probe screen works as a triage signal. Its unadjusted association with conditional collapse risk, measured at the level of model families, is high: a Spearman rho of zero point eight nine three. But the accuracy of the initial majority is a close comparator, at zero point eight two one, so the authors decline to treat the screen as calibrated prediction, or as prediction adjusted for capability. Round-level traces place many collapses in the first debate round, where early disagreement can come before harmful cascades and useful recovery alike.

The release is built for comparison: replayable schemas, coders, audits, cost cards and rebuild scripts that need no API calls, so future rows pairing a model with a scaffold can be scored under the same denominators and the same signed ledger. Underneath the tooling is a bookkeeping rule. An intervention should be credited for the collapses it stops and charged for the rescues it blocks.