Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions
Yucheng Du, Xiyang Hu
University of Southern California · Arizona State University
Published August 29, 2026 · on the feed September 9, 2026 · ▲ 17 · arXiv 2608.29109
The abstract
Large language models often answer structurally unanswerable questions, such as computing cot(-540°) or evaluating (1).startswith("1"), instead of abstaining. We ask whether this failure reflects missing recognition or failed routing from recognition to abstention. Across instruction-tuned models from 1.7B to 70B parameters, a single linear direction in the hidden state separates answerable from structurally impossible math and code prompts, showing that models represent impossibility before generation. Yet this recognition direction is nearly orthogonal to the canonical safety-refusal direction that mediates trained harmful-content refusal. An in-domain behavior-defined invalidity-aware direction is closer to recognition, but only partially aligned with it, and remains near-orthogonal to safety refusal. Generation-time steering along the recognition direction changes invalidity-aware behavior bidirectionally and dose-responsively on structural math and code cells, while random directions do not. Base/instruct comparisons further show that the low-cosine geometry is already present at the pretraining endpoint. The confident-on-impossible failure is therefore better explained as a routing failure than as an encoding failure: the model has a usable "no admissible answer" signal, but the safety-refusal pathway is not aligned to use it.
The Brief
2:34A short podcast summary of the paper. Written and voiced by AI.
Transcript
Recognition-Refusal Misalignment in LLMs. Why Models Answer Structurally Unanswerable Questions. By Yucheng Du and one other, from the University of Southern California and Arizona State University. Published on the twenty-ninth of August. Its code repository is worked out: about forty source files, with a readme and a licence.
Ask a large language model for the cotangent of minus five hundred and forty degrees, or to evaluate a snippet calling startswith on a parenthesised one, and it will often give you an answer. These are structurally unanswerable questions, and the model should abstain. The question the authors put is why it does not. One possibility is that the model never recognises the impossibility. The other is that it does recognise it, and the recognition simply fails to reach the behaviour of abstaining. Their conclusion is the second: this is a routing failure rather than an encoding failure. The model holds a usable signal meaning no admissible answer exists, but its safety- refusal pathway is not aligned to use it.
The evidence comes from the geometry of hidden states. In instruction-tuned models ranging from one point seven billion to seventy billion parameters, a single linear direction separates answerable prompts from structurally impossible math and code prompts. The models represent impossibility before they generate anything. Yet that recognition direction sits nearly orthogonal to the canonical safety-refusal direction, the one that mediates trained refusal of harmful content.
A third direction, defined in-domain from invalidity-aware behaviour, lands closer to recognition but only partly aligned with it, and it too stays near- orthogonal to safety refusal. The causal test is steering. Pushing generation along the recognition direction shifts invalidity-aware behaviour in both directions and in a dose-responsive way on the structural math and code cells. Random directions produce no such effect. Comparing base and instruct models shows that the low-cosine geometry is already there at the end of pretraining.
The claim is bounded by where it was measured. The tasks are structural impossibilities in math and code, not unanswerable questions in general, and the steering results are reported on those structural cells specifically. The models are instruction-tuned models in the stated size range, with base models used for the pretraining comparison. What is established is a representational gap between recognising and refusing within that setting.
The diagnosis moves the problem to a different place. If models failed to encode impossibility, the remedy would be teaching them to notice. Since the signal is already present and steerable, the missing piece is the connection between it and the refusal machinery that training built for a different purpose.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A model calmly answers a question that has no answer. Is it blind, or is something not wired up?
You ask a chat model for cot(-540°), which is undefined, or to evaluate (1).startswith("1"), which raises an error. It confidently answers anyway, instead of saying there is no answer.
The road ahead
- 1 · The problem ~7 min Chat models confidently answer questions that have no answer, and you must work out whether they fail to notice or fail to act.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min The model does recognise impossibility inside, but that signal sits on a different axis from safety refusal, so it never reaches declining.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build matched pairs, find the direction that splits them, compare it with safety refusal, steer along it, then check base models.
- 6 · The experiment ~6 min You’ll guess: Models hold a hidden-state direction that separates answerable from impossible math and code prompts. How does it relate to the direction that drives safety refusal?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.