You Changed Your Mind, The Model Didn't: Demystifying Intent in Multi-Turn Dialogue

You Changed Your Mind, The Model Didn't. Demystifying Intent in Multi-Turn Dialogue. By Junle Chen and seven others, from the Hong Kong University of Science and Technology and Tencent HY. Published on the fifth of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Picture a multi-turn task with a language model. Partway through, the user proposes a change, thinks it over, and rejects it. The right behaviour is obvious: carry on as if nothing changed. The authors report a surprising failure here. Merely mentioning a change that is later rejected can derail task execution, even though the user's final intent is exactly what it was before. Studying this needs a controlled way to vary what a user says without varying what the user wants.

Intent-Eval is that benchmark. It is built to study how language models behave when user intent evolves over a conversation, and it isolates two situations: proposals that the user rejects, and requirements that a later request supersedes. In both, the measure is whether the model keeps following what is actually in effect.

The benchmark is controlled and spans four kinds of work: tool actions, code, databases, and mathematics. Across these diverse tasks, the same pattern can be tested in quite different settings, from calling tools to solving problems.

Models prove vulnerable to both rejected proposals and superseded requirements. The authors describe the pattern as mentioned-as-in-effect confusion: conversational content is treated as an active requirement even after it has been rejected or replaced. The damage does not simply fade. Accuracy degradation can deepen or persist as the interaction continues. What is needed, the authors argue, is for models to tell apart what has been mentioned from what remains in effect.

Building on that diagnosis, they propose Intent-OPSD, a decision-conditioned on-policy self-distillation framework. Teacher and Student start from the same model. The frozen Teacher sees the complete task matching the user's final decision, and supervises the Student on the full dialogue, so the Student learns to follow the requirements that reflect active intent. A conversation, in this framing, is a record of what was said, and only part of it is an instruction.