NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

NCP-ArchPreview Technical Report. Moving towards Latent Space Language Models through Next Concept Prediction. By NCP Team and twenty-seven others, from the Shanghai AI Lab and the LUMIA Lab at Shanghai Jiao Tong University. Published on the ninth of September. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.

Autoregressive pretraining teaches a model one thing at every step: predict the next token. NCP-ArchPreview asks what happens if a model is also made to predict something bigger. Its authors describe it as a move toward latent-space language models, models that learn to predict in a space of concepts rather than only in the space of tokens. Next-token prediction works, but it asks the model to look only one small step ahead. The question behind this report is whether a model that is also asked to anticipate larger units of meaning, several tokens at a time, learns more from the same data.

Alongside standard next-token prediction, NTP, the model learns through Next Concept Prediction, NCP. A concept here is a discrete unit that spans multiple tokens. Predicting concepts adds an explicit objective at the concept level, one that is more challenging, while generation itself stays standard, autoregressive, and token by token. The design keeps what already works and adds to it. Nothing about how the model produces text changes from the outside. The change is in what it is asked to learn while training.

The latent space is built from the model's own hidden states, which are used to construct a product-quantized vocabulary of concepts. A dedicated Concept Module then learns to predict future concepts. Those predictions are fed back to the token level to guide the generation that follows. NTP and NCP are trained jointly, end to end. The loop runs in both directions: tokens give rise to concepts, and predicted concepts in turn shape the tokens that come next.

The architecture was scaled to eight point nine billion parameters and trained on five point seven three trillion tokens from the Dolma-3 dataset. The authors call it the largest demonstration of a latent-space language model to date.

Against OLMo-3-7B, NCP-ArchPreview reaches the same final pretraining loss after consuming only fifty-one point three percent of the total training tokens. After full pretraining, it outperforms OLMo-3-7B by two point four five points on the downstream macro-average, including a gain of five point nine nine points on GSM8K. Put another way, it gets to the same place on roughly half the data. And when trained on all of it, it ends up ahead.

Controlled experiments isolate gains from both ingredients, the latent architecture and the NCP objective. Using only eighty-five percent of the standard computation, the model approaches the training loss of an eight point nine billion parameter baseline whose parameters are strictly aligned with its own.

The latent space keeps its value after pretraining ends. Updating only the VQ module, with seventeen million parameters, gives a new and lightweight interface for domain adaptation. And simply injecting concept representations into a DFlash2 drafter raises the mean accepted length by four point one seven percent, with negligible overhead. Concepts, then, are not only a training signal. They are a handle on the model that can be used afterwards.

NCP-ArchPreview Technical Report. Moving towards Latent Space Language Models through Next Concept Prediction. By NCP Team and twenty-seven others, from the Shanghai AI Lab and the LUMIA Lab at Shanghai Jiao Tong University. Published on the ninth of September. This is the long version: the whole paper rather than the abstract.

Standard next-token prediction supervises a language model one token at a time. Models do form higher-level abstractions in their hidden states, but only as a byproduct: nothing in the objective says how meaning should unfold across a span of several tokens. Next Concept Prediction, introduced by the earlier ConceptLM work, adds that objective. ConceptLM trained models from scratch only up to one point five billion parameters, and otherwise bolted the idea onto an existing model by continual pretraining. The question here is whether it survives the full treatment: an eight point nine billion parameter model, built on the OLMo-3-7B architecture, with concept prediction switched on from the first step and trained on five point seven three trillion tokens of the exact data OLMo-3-7B saw.

Two pieces of machinery come first. The first is pooling: the model averages every four consecutive token states into one continuous concept vector, so the concept sequence is a quarter the length of the token sequence. The second is product quantization, which gives those concepts a vocabulary. Each concept vector is cut into thirty-two segments, every segment has its own small codebook of one hundred and twenty-eight entries, and each segment snaps to its nearest entry. Small codebooks, but the number of possible combinations is one hundred and twenty-eight raised to the thirty-second power. The codebooks learn through a loss that only pulls the chosen entries toward the concept vectors; a stop-gradient keeps that loss from reshaping the token states themselves.

The network is split into three modules: a sixteen-layer Token Encoder, an eight-layer Concept Module, and a sixteen-layer Token Decoder. The encoder's states are pooled into concepts, and the Concept Module reads the history of concepts and, for each of the thirty-two segments, outputs a probability distribution over that segment's codebook. The expected codeword under each distribution, stitched back together, is the predicted next concept. Its loss is a squared distance to the true next concept, with the target detached, so the encoder learns only through the history it supplied: it is pushed to keep whatever helps predict what comes next.

Each predicted concept is then repeated four times to token resolution, shifted so that it arrives only after the tokens used to predict it have been processed, and added directly to the encoder's token states. The decoder predicts the next token from that sum, so generation remains ordinary token-by-token decoding. On top sit learned residual connections, adapted from MUDDFormer, that let each layer mix states from earlier depths and pass information between the three modules. They are initialized to behave like plain residuals, so training starts close to the original backbone. The total loss is next-token prediction plus weighted concept and codebook terms, all trained end to end.

Against OLMo-3-7B on identical data, the first-stage loss gap widens through training and ends at zero point zero nine one. The concept model reaches the baseline's final loss after fifty-one point three percent of the tokens, which is the one point nine five times speedup in the abstract. Downstream, the macro-average rises by two point four five points.

The comparison that actually tests the thesis is quieter. A Concept Module block holds about as many parameters as an ordinary block but costs about a quarter of the compute, since its sequence is a quarter as long. So the authors built two extra baselines: a thirty-four-layer model matching the compute, and a forty-layer model matching the parameters. Over the first two hundred billion tokens, the concept model clearly beats the compute-matched one and approaches the size-matched one on eighty-five percent of its compute. Adding the pieces one at a time, the Concept Module, then the residual routing, then the concept loss, each lowers the loss further.

The paper has a short limitations section with two items. First, there is no long-context training yet: everything runs at a maximum of eight thousand one hundred and ninety-two tokens. Second, lower loss does not reliably become better scores. After the second stage the gain shrinks to zero point five nine points, and the code average actually falls, which the authors attribute to that stage's data being only about ten percent code. Three second-stage recipes showed progressively lower loss and progressively worse downstream results. Two further caveats come from the body. The residual routing's costs are counted in analytical operations, which the authors note exclude memory traffic and kernel launch overhead. And the main runs kept OLMo's layer-wise query and key normalization even though it caused exploding attention logits under the Muon optimizer, precisely to keep the comparison with OLMo controlled.

The idea to carry away is that a model's own latent space can be a prediction target at trillion-token scale while the output stays plain tokens. The learned concept vocabulary also turns out to be a handle afterwards: tuning just its seventeen million parameters adapted the model to mathematics while its general average held steady, and feeding concepts into a speculative drafter raised mean accepted length by four point one seven percent.