BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion

BYOD. Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion. By Ruurd J. A. Kuiper and one other, from the University Medical Center Utrecht and Utrecht University. Published on the thirtieth of October.

Diffusion language models can do things autoregressive models cannot. They update tokens in parallel, generate flexibly, and attend in both directions. But the leading systems get there through costly pretraining or by fine-tuning the entire model. Either way, bringing diffusion generation to a model family that already exists is expensive.

Build Your Own DLM, or BYOD, is a recipe for converting an autoregressive model into a diffusion language model using nothing but LoRA adapters. It is parameter-efficient and meant to generalize, with no tuning specific to any one architecture.

The base model is left untouched. Every one of its parameters stays frozen, and the adapters make up four point one to five point eight percent of the final parameter count. Each conversion runs for twenty-five thousand iterations and needs at most one hundred and two point four million tokens. That takes between three point one and four hours on a single Google Colab GPU.

The recipe is applied to four models: Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B. To the authors' knowledge, the last of these is the first reported conversion of a Mistral model from autoregressive to diffusion.

All four converted models generate from fully masked sequences. All four use bidirectional context in a controlled attention test. And all four keep substantial knowledge from their parent models. The strongest conversion on a nine-task evaluation is BYOD-Gemma, with an average accuracy of fifty point six percent. In the same harness, LLaDA 8B Instruct reaches fifty-one point nine percent.

The authors are direct about the main limitation: the quality of open generation when the budget of network function evaluations is low. Generating one hundred and twenty-eight tokens in sixty-four steps, BYOD models reach a perplexity between eight point three and eleven point one. LLaDA reaches four point one. The autoregressive parents reach between two point two and three.

To support further experiments and inspection, the authors release a standalone training and evaluation implementation, along with interactive demos for all four conversions.

The trade BYOD offers is plain. For a few hours on one GPU and a small set of adapters, an existing autoregressive model gains diffusion generation and comes close to a dedicated diffusion model on accuracy. What it does not yet match is open-generation quality at low step counts, and the authors name that gap as the one efficient conversion has to close next.

BYOD. Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion. By Ruurd J. A. Kuiper and one other, from the University Medical Center Utrecht and Utrecht University. Published on the thirtieth of October. This is the long version: the whole paper rather than the abstract.

Diffusion language models do not write left to right. They start from a block of masked positions and fill many of them at once, over a number of steps you choose, using context on both sides, and early guesses can be revised. The catch has been the bill. Under the paper's standardized rental scenario, LLaDA's from-scratch pretraining, on two point three trillion tokens and a hundred and thirty thousand GPU hours, comes to roughly six hundred and seventy-five thousand dollars. Converting an existing autoregressive model is cheaper, but earlier conversions still updated every weight over tens or hundreds of billions of tokens. The question here: can one small, fixed recipe convert chat models from four families in a few hours on a single GPU, and what is lost along the way?

Two ideas carry the rest. The first is masked diffusion. During training, a masking rate is drawn somewhere between almost zero and one, each answer token is hidden independently with that probability, and the model must recover the originals. Generation runs the process the other way. It begins from an answer block that is entirely masks, predicts every masked position, keeps the most confident predictions, re-masks the rest, and repeats. The number of these network evaluations, called NFE, is set independently of the output length. The second idea is LoRA: the original weights stay frozen, and small low-rank matrices are trained alongside chosen projections. On top of those sits the study's design. Gemma, Llama, Qwen and Ministral, all instruction-tuned and all between seven and nine billion parameters, receive the identical intervention, and each converted model is scored against the exact checkpoint it came from.

The conversion makes three small changes. First, the causal attention mask in every layer becomes bidirectional, so every position can attend to every other. Second, the label shift is removed. An autoregressive model's output at each position predicts the next token. BYOD keeps the pretrained output head untouched and trains it to predict the clean token at the same position instead. Even the mask token is borrowed: each model reuses a string its tokenizer already has, so the vocabulary never grows. Third, LoRA adapters go on every query and value projection, at a rank of one thousand and twenty-four in all four families. Everything else stays frozen.

The prompt is never corrupted and never scored. Only the response and the end-of-sequence padding after it can be masked, with the loss weighted nine parts answer to one part padding. The first end-of-sequence token counts as a genuine answer token, which is how the model learns where an answer stops. The data mixes instruction following, multiple-choice reasoning, mathematics and code, and each conversion runs twenty-five thousand updates on one GPU in Google Colab, in three point one to four hours. Fixing the rank keeps one recipe for every architecture, but gives Gemma the largest absolute adapter. Afterwards the adapter can be folded into the original weights, restoring the original model size.

Every model, including LLaDA, is scored in the same harness, on the same fixed subset of a hundred and twenty-five examples from each of nine benchmarks. The best conversion is Gemma's, averaging fifty point six percent against LLaDA's fifty-one point nine. The parents still do better on average. The comparison that most directly tests the paired design is Qwen: the strongest parent of the four, yet its conversion lands below Gemma's. Parent strength alone does not decide conversion quality. The losses concentrate in mathematics and code, and the full paper shows part of why. Of the converted Llama's HumanEval programs, twenty-eight percent fail Python's parser, against none for its parent: a duplicated keyword here, a missing parenthesis there.

The bidirectional test is a copy task. A word appears later in the response, and the model must fill an earlier masked slot with it. Adding the clue changed the prediction on every item, for every model. Copying it exactly was another matter: Gemma and Ministral managed it every time, Qwen only a fifth of the time. Doubling Llama's run to fifty thousand updates kept validation loss falling, but the average rose by half a point, with five tasks up and four down. Fluency is the clearest gap. Writing a hundred and twenty-eight tokens in sixty-four steps, the conversions score a perplexity between eight point three and eleven point one, against four point one for LLaDA and three or below for the parents.

The paper has a limitations section, and its first point is the important one: efficient conversion does not yet produce efficient generation. Output quality trails both the parents and LLaDA, especially when the step count is below the output length. Budgets are counted in network evaluations, not wall-clock time, and since each diffusion step processes the whole answer block, fewer steps do not guarantee a speedup. The efficiency claim is about training the conversion, nothing more. The cross-family comparison is not a controlled architecture study, so Gemma's lead cannot be attributed to architecture, parent quality, size or adapter budget. Beyond what the authors list: each benchmark is a subset of a hundred and twenty-five examples, no error bars are reported, and LLaDA itself scored two point six points below its published average in this harness. And while one conversion takes hours, developing the recipe took hundreds of GPU hours.

The idea to carry away is that diffusion behaviour is cheap to induce and expensive to make good. With every original weight frozen, bidirectional attention arrives in all four families, and much of the parent's knowledge survives, most visibly on multiple choice. What does not come along is fluent, syntactically clean generation at small step budgets. The authors' proposed next step: start fuller diffusion fine-tuning from the merged model, with diffusion generation already established before the original weights are exposed. Meanwhile the recipe is a cheap, repeatable instrument for asking how the choice of parent shapes a diffusion model.