Unmask the State: When Does State Adaptation Matter for Masked Diffusion Language Models

Unmask the State. When Does State Adaptation Matter for Masked Diffusion Language Models. By Injin Kong and two others, from Seoul National University. Published on the twenty-seventh of September. No code repository is linked to it.

Masked diffusion language models, MDMs, can generate in flexible orders, which turns the unmasking strategy into a decision made at inference. Existing methods differ in how they prioritize positions, control parallelism, restrict where selection happens, revise predictions, or plan future denoising. What stays unclear is when those choices ought to change partway through generation. A fixed strategy holds the same choice from the first token to the last. The open question is whether, and where, switching would pay off.

The authors approach this through strategy reversals: moments where some alternative action becomes better than a fixed choice. They organize MDM inference along five axes, namely score, cardinality, region, commitment, and planning. Adaptation opportunity is then defined as the one-step utility advantage of the best candidate action over a fixed action chosen on validation data. Seen this way, the value of adapting depends on two things: how often reversals happen, and how large they are. Rare but large reversals and frequent but tiny ones are different situations, and a single average would hide the difference.

Across three MDMs and ten tasks, adaptation opportunities turn out highly heterogeneous. Some regimes show one-step gains that are concentrated and predictable. That points to selective adaptation. Lightweight detectors, calibrated on validation prompts, pick out the high-opportunity states. In one example, LLaDA-8B on constrained JSON filling, adapting only the top ten percent of states captures fifty-six point nine percent of the candidate-set oracle opportunity. A tenth of the decisions, in other words, accounts for more than half of what an oracle choosing among the candidates could have gained in that setting.

These are transition-level results, built on a one-step measure of utility and on a candidate set of actions, so the oracle being approached is the best of those candidates. Within that frame, they suggest that state adaptation is most useful when applied selectively rather than everywhere.

Picture adaptive decoding as a switch one could leave on all the time. The evidence here says its value is uneven from state to state. Where gains are concentrated and predictable, a cheap detector can find them, and those states are where adapting belongs.