System Switch: When Should a Fast Decision Model Stop and Think?

System Switch. When Should a Fast Decision Model Stop and Think?. By Gian Luca Bailo, working independently. Published on the seventh of October.

Dual-process agents pair a fast policy with a slow, deliberative model. In real-time settings the slow model usually runs all the time. In turn-based agents and robot planners it is called on events, such as uncertainty or a detected failure. This study looks at a third arrangement. A fast learned actor takes every decision, and hands control to a reasoning vision-language model only when a gate opens, while the game keeps running. The question in the title is when that gate should open, and the answers turn out to depend on something other than raw accuracy. The setting is closed-loop Doom, with the new open System One typed-decision models, served through a common llama.cpp interface.

The first results come from nine hundred held-out questions. Zero-shot decision models, ranging from zero point one five billion to nine billion parameters, lean towards collecting items when they are wrong: among their errors they choose to collect items one point six to one point eight times more often than chance. That holds in any option order, though the order does change some models' accuracy. Second, accuracy, calibration and sensitivity are distinct things. Sensitivity here means how well confidence separates right answers from wrong ones. Models of similar accuracy differ widely in AUROC, and the confidence of the most sensitive model tracks which kinds of situation it fails in, not which particular answers are wrong.

Third, deferral. Offline, handing the least confident thirty percent of decisions to a reasoning model beats random deferral, and the gain grows in proportion to the actor's AUROC, with a rank correlation of zero point eight seven. When the actor and the deferral rate are chosen on held-out games, the gain is zero point one three, with an interval from zero point zero eight to zero point one eight, using doomLaya's option order. With shuffled options it is zero point zero eight, with an interval from zero point zero two to zero point one four. Reasoning carries about half of that gain.

The closed-loop results set the limits. Across thirty-three games and three seeds, no variant reaches the exit. Committing to plans, whether the reasoner's or a fixed explore rule's, opens more doors, and turns an actor that would otherwise stand still into one that plays. With the fixed rule, though, the agent dies more often. And background knowledge cuts both ways. Told that some doors need keys, the reasoner mistakes ordinary doors for locked ones, because the state cannot tell the two apart. Without that knowledge, it goes back to collecting. So the offline gains from confidence gating do not yet carry through to finishing a level.

What the study leaves behind is a separation between three properties people often treat as one. A good gate needs an actor whose confidence ranks its own mistakes, and that is a different thing from being accurate or calibrated. The author releases the code, prompts, data and logs, so the gap between the offline gain and the closed-loop outcome can be examined directly.