Hierarchical Continuous Diffusion Language Models
Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing
University of Illinois Urbana-Champaign · Amazon.com, Inc.
Published October 1, 2026 · on the feed October 2, 2026 · ▲ 82 · arXiv 2610.02193
The abstract
Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.
The Brief
2:28A short podcast summary of the paper. Written and voiced by AI.
Transcript
Hierarchical Continuous Diffusion Language Models. By Hui Ren and four others, from the University of Illinois Urbana-Champaign and Amazon. Published on the first of October. Its code repository is a placeholder: a readme, and nothing to run yet.
Discrete diffusion language models are an alternative to autoregressive generation that suits tasks needing bidirectional reasoning and global constraint satisfaction. They share a structural bottleneck, though. When they decode several tokens in parallel, each one is sampled independently from its own marginal distribution, which severs the statistical dependencies among tokens decoded together. Continuous diffusion language models avoid that by denoising one shared continuous state. Their weakness sits on the other side: the denoiser sees only that continuous state, and nothing ties it to a valid configuration of tokens until the very end, when it is finally decoded. One family keeps tokens real but loses the links between them. The other keeps the links but floats free of real tokens.
Hierarchical Continuous Diffusion Language Models, or HC-DLM, couple the two. Discrete token generation and a continuous latent trajectory are joined inside a single denoising process, and its training objective is derived from a variational bound on the likelihood of the tokens. The distinguishing choice is which state persists. Recent methods attach continuous context to a discrete chain that stands on its own. In HC-DLM, the latent is the only persistent generative state. The tokens are not a second chain running alongside it. They are a view of it, taken again at every step.
At every step, tokens are read out from the latent. Those tokens then feed back as a scaffold for the next update of the latent. The continuous state keeps carrying the joint information, and the discrete readout keeps it anchored to actual tokens at each step, rather than only once at the end.
Three tasks are used, and comparisons are against discrete and continuous diffusion baselines at matched model size. On structured reasoning with Sudoku and on mathematical planning with Countdown, HC-DLM improves puzzle accuracy. On language modelling with LM1B, it improves generative perplexity. The two puzzles are the kind of task where a valid answer depends on many positions agreeing at once, which is the regime the problem statement was about. The abstract reports the direction of each improvement, not its size.
Each family of diffusion model had a missing piece the other could supply. Making the latent the one thing that persists, and reading tokens out of it at every step, is how HC-DLM tries to keep both.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Puzzles punish any answer whose parts disagree. Here is the wall parallel decoding runs into.
You are building a diffusion language model to solve Sudoku and arithmetic puzzles, where a valid answer needs many positions to agree at once.
The road ahead
- 1 · The problem ~6 min Puzzle answers need many tokens to agree, but one kind of diffusion model samples tokens separately and the other never checks real tokens until the end.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Keep one shared continuous latent as the only lasting state, read tokens out of it every step, and feed them back to guide its next update.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Generation loops: begin with a noisy latent, read tokens out of it, feed them back as a scaffold, and update the latent with that scaffold.
- 6 · The experiment ~6 min You’ll guess: On Sudoku and Countdown, how did HC-DLM's puzzle accuracy compare with discrete and continuous diffusion baselines of matched model size?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.