Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

Cola2. Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation. By Hongcan Guo and six others, from the University of Hong Kong, ByteDance Seed, and two other institutions. Published on the twenty-second of October. No code repository is linked to it.

Hierarchical language generation splits the job in two. Meaning is modelled in a continuous space, and the actual text is realised discretely afterwards. Whether that works depends on two things: what the latent space compresses, and how the discrete generator makes use of it. A latent that keeps the wrong information, or a generator that ignores it, undoes the point of separating the levels.

Cola2 is a framework that studies three questions together: continuous semantic compression, the balance between reconstruction and generation, and co-adaptation between the continuous and discrete parts. It dynamically segments text into semantic spans. It controls how hard those spans are compressed through auxiliary-loss-free negative feedback. And it combines reconstruction and generation supervision, so the latent representations it learns are both compact and predictive.

On the discrete side, a block-causal masked diffusion decoder is trained under three latent conditions: clean, noisy, and dropped. Classifier-free guidance then adjusts the balance between reconstructing the semantics and generating autonomously. Gradients from token prediction flow back through the latents to update the encoder, while the Flow Matching prior is kept separate from representation learning by a stop-gradient. At inference time, generated text is re-encoded under a fixed span layout, so the latent history stays consistent with the text that was actually produced.

Each of these choices maps onto one of the three questions. The dynamic spans and the negative feedback govern compression. The mix of reconstruction and generation supervision, and the guidance at decoding, govern the balance. And the gradient path from token prediction into the encoder is where continuous and discrete parts adapt to each other, while the stop-gradient keeps the prior's learning from interfering with the representation.

Experiments on five downstream tasks, with about two hundred and ten billion training tokens, validate the core designs and show competitive scaling that has not yet saturated. A larger run used three hundred and forty-six billion source tokens.

At that scale, a checkpoint that has not converged reaches an average of sixty point one across eight tasks. It surpasses Llama-3-8B on HellaSwag and StoryCloze, even though that model's reported pretraining budget is over forty-three times larger.

The authors present these results as showing the potential of continuous-latent hierarchical generation. The design has many parts, but they answer the two conditions set out at the start: spans and feedback decide what the latent compresses, and the decoder's training conditions and guidance decide how the discrete generator uses it.

Cola2. Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation. By Hongcan Guo and six others, from the University of Hong Kong, ByteDance Seed, and two other institutions. Published on the twenty-second of October. This is the long version: the whole paper rather than the abstract.

An earlier model from the same group, Cola DLM, split writing into two levels: plan the meaning in a continuous latent space, then realize that plan as discrete tokens. The authors' complaint about that split is precise. Having two spaces does not by itself give you a sensible division of labour. Shortening the latent sequence is not the same as compressing meaning, and a latent trained only to reconstruct its own text need not help predict the text that comes next. Meanwhile the token decoder has to cope when the plan it receives is incomplete or wrong. Cola2 is their attempt to decide, jointly, what the latent should carry and what the decoder should be left to fill in.

Three modules carry the design. An encoder compresses a stretch of text into a shorter sequence of continuous vectors. A latent generator, a Diffusion Transformer trained with Flow Matching, learns to produce those vectors from context. A decoder turns each vector back into tokens. The paper frames this as an accounting of uncertainty: part is explained by the latent, and the residual is left for the decoder. Change the encoder and you move that boundary. The compression rate is simply latents per token: one latent for every two tokens is a rate of one half. A good latent, by the authors' definition, packs high-level semantic information densely into those few positions, and reduces uncertainty about missing or future tokens, not only the tokens it was built from.

Start with segmentation. Instead of cutting text into fixed chunks, the encoder measures how sharply the token representations shift between neighbours and places a span boundary where the shift crosses a threshold, with a hard cap on span length. A single bias term sets how readily boundaries appear, and it is not learned by gradient at all. After each batch, if the measured compression rate is above target, the bias is nudged one way; if below, the other. That is the auxiliary-loss-free negative-feedback controller, and the appendix proves it converges locally while the encoder is held fixed. Within each span, tokens attend to each other in both directions and are pooled into one latent.

The decoder is a block-causal masked diffusion model: causal across blocks, denoising tokens in parallel inside the current block. Its training is where the balance between reconstruction and generation gets built in. For each block it sees the latent in one of three states: clean, mixed with noise, or dropped entirely. Clean teaches faithful realization; noisy and dropped force it to generate from context alone. A second objective masks part of the encoder's own input and asks the system to predict those hidden tokens, pushing the latent toward predictive content. Then gradient routing. Token prediction loss flows back through the latents into the encoder, so the representation adapts to what the decoder actually needs. The Diffusion Transformer's loss is stopped before it reaches the encoder. At inference, each generated token block is re-encoded under the span layout already chosen, and that corrected latent, not the original proposal, is what enters the history.

Most ablations score LAMBADA accuracy at small scale. Dynamic segmentation at a rate of one half reaches twenty-nine point four percent with a looser span cap, slightly above the uncompressed model at twenty-eight point seven. The feedback controller scores twenty-five percent against eighteen point four for DLCM's auxiliary rate loss. Generative supervision of the encoder lifts pure reconstruction from fifteen point seven to twenty-six point five. Freezing the encoder against co-training it: forty-seven point six against fifty-eight point two.

The most direct test of the gradient routing thesis is a time series. Letting the generator's gradient into the encoder leads by two point two points early in training, then falls behind, ending one point down. A shortcut that looks good early is exactly what latent hacking predicts. The decoder clearly uses the latent: shuffling latents, which keeps their statistics but breaks their meaning, drops accuracy from forty-six point five to thirty-four point three. In controlled scaling to about two hundred and ten billion tokens, the compressed model matches the autoregressive baseline on LAMBADA and leads on RACE, but loses ground on SQuAD and Story Cloze, which reward verbatim reproduction. The uncompressed variant scales best. The headline checkpoint averages sixty point one over eight tasks, against sixty-six point two five for Llama-3-8B, and beats it on HellaSwag and StoryCloze.

The paper has a short limitations section, and it is candid. The model is far from convergence, especially the latent generator, and training was limited by available resources. Compression cuts the theoretical cost of the latent prior, but end-to-end speed also depends on the decoder, guidance branches, feedback passes and hardware. The cost analysis offers only a call-round proxy and says plainly that it is not a latency guarantee; no measured speedup is reported. The best division of labour between autoregressive and masked diffusion decoders is called underexplored. Beyond that list, the final checkpoint departs from the studied recipe: its decoder keeps causal self-attention, and a third stage freezes the encoder entirely. Component results are single LAMBADA numbers without error bars.

Carry away the routing rule. In a two-level language model, the encoder should be shaped by what the token decoder needs, never by what the latent generator finds easy. Train the decoder mostly on degraded plans so it can fill gaps, keep the latent history honest by re-encoding what was actually written, and let a simple feedback loop, not a loss, hold the compression rate.