Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?
Minghan Wang, Boyuan Wang, Jinhang Zuo, Yuxin Tao, Fang kong
Southern University of Science and Technology · City University of Hong Kong
Published September 30, 2026 · on the feed October 1, 2026 · ▲ 61 · arXiv 2609.39578
The abstract
Agent harnesses often improve language models with human-designed workflows, but as models grow more capable, unreliable guidance can increasingly constrain their execution. We call the ability to benefit from useful guidance while overriding unreliable guidance thinking outside the box. We introduce Box^2-Bench, which holds the model and task fixed while varying workflow reliability to isolate how models regulate their reliance on guidance. On Box^2-Bench, frontier models often benefit from reliable guidance but remain vulnerable when it is misleading or becomes unreliable. To test whether this capability can be learned, we train two open-weight models using bad workflows, reserving good workflows for evaluation. We explore two complementary training strategies: counterfactual supervised fine-tuning improves robustness, while outcome-based reinforcement learning can shift the balance toward greater use of helpful workflows. We further find that this behavior extends beyond workflows to other forms of external information, improving peer correction and robustness to corrupted memory. Together, our results identify selective reliance on fallible external information as a dimension of agent reliability not captured by task performance alone.
The Brief
1:59A short podcast summary of the paper. Written and voiced by AI.
Transcript
Thinking Outside the Box. Can Language Models Rely on External Guidance Selectively?. By Minghan Wang and four others, from the Southern University of Science and Technology and City University of Hong Kong. Published on the thirtieth of September. No code repository is linked to it.
Agent harnesses often lift a language model's performance by giving it workflows that people designed. As models become more capable, though, guidance that is unreliable can increasingly hold their execution back. The authors name the skill that matters here thinking outside the box: taking the benefit of guidance when it is useful, and overriding it when it is not. Their claim is twofold. Frontier models lack this skill in an important way, and it can be learned.
The measuring instrument is Box squared Bench. It keeps the model and the task fixed and varies only how reliable the workflow is, which isolates how a model regulates its reliance on guidance. On it, frontier models often profit from reliable guidance but stay vulnerable when the guidance is misleading or turns unreliable. To see whether selective reliance can be trained, the authors take two open-weight models and train them on bad workflows, holding good workflows back for evaluation. Two strategies complement each other. Counterfactual supervised fine-tuning improves robustness. Reinforcement learning on outcomes can tip the balance toward making more use of workflows that help.
The behaviour reaches past workflows. Training for it also carries over to other kinds of external information, improving how models handle correction from peers and how robust they are to memory that has been corrupted. The training itself used workflows only, so these are reported as the same behaviour extending outward, not as separate skills.
Task accuracy alone would miss all of this. A model can score well while following good advice blindly, and the same habit is what sinks it when the advice goes bad. The authors place selective reliance on fallible outside information as its own dimension of agent reliability, one that has to be tested by changing the guidance and holding everything else still.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Human workflows help models, until the model is better than the workflow. Here is the bind.
You build a harness that hands a language model a step-by-step workflow written by people. When the steps are right, the model often does noticeably better.
The road ahead
- 1 · The problem ~6 min Human-written workflows help a language model when they are right, but they drag a stronger model into failure when they are wrong.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Measure reliance by changing only the workflow's reliability, then train on misleading workflows with fine-tuning and outcome-rewarded reinforcement learning.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build matched workflow conditions, train only on misleading workflows with fine-tuning then reinforcement learning, and test on good workflows never trained on.
- 6 · The experiment ~6 min You’ll guess: When frontier models on the new benchmark are handed workflows that mislead them, how do they hold up?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.