SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation
Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss
Stability AI · Karlsruhe Institut für Technologie
Published September 30, 2026 · on the feed October 2, 2026 · ▲ 11 · arXiv 2610.00686
The abstract
Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4times its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
SemanTok. Predictable Semantic Tokens for Efficient Autoregressive Video Generation. By Mikhail Dereviannykh and five others, from Stability AI and the Karlsruhe Institute of Technology. Published on the thirtieth of September. Its code repository is public and real: about twenty source files.
Recent video world models pair autoregressive prediction, for its scalability, with diffusion models, for their visual quality. For both halves, the choice of scene tokenizer matters a great deal, in fidelity and in semantics. Flexible-length tokenizers that work coarse to fine deliver both: the first coarse tokens carry the global semantics of a clip, and later tokens specify the details. The weak point is how the semantics get learned. Existing flexible tokenizers apply a representation-alignment loss, REPA, only on early hidden states of the decoder. That is a target the decoder can partly meet from its noised input instead.
SemanTok puts the semantic signal on the tokens. It is a flexible video tokenizer that feeds frozen DINO features into its encoder, and adds lightweight heads that reconstruct those features from each retained token prefix alone, with nothing else to lean on.
Because each reconstruction works from a prefix by itself, every retained prefix is asked to carry those features on its own. Short prefixes then hold the semantics, they are cheaper for an autoregressive model to predict, and pixel detail is deferred to the later tokens.
SemanTok achieves high semantic alignment and video fidelity at every autoregressive model size tested. A SemanTok AR model at two hundred and one million parameters matches or beats a VideoFlexTok AR model three point four times its size, and larger SemanTok AR models improve fidelity further. Semantic alignment holds on out-of-distribution classes, and the decoder gets higher semantic alignment at every noise level, including pure noise.
It performs well at both reconstruction and generation, the two jobs a tokenizer has to serve. Its short token prefixes are cheaper to predict, and they also give better generation fidelity, with pixel detail left to the later tokens. Cheaper and better here come from the same short prefix, which fits the coarse-to-fine design the whole approach starts from.
Where a loss is attached decides what it ends up shaping. On the decoder's early states, the semantic target could be met partly from noise. On every token prefix, it shapes the very tokens an autoregressive model has to predict, and that is where the efficiency in the title comes from: a smaller model, and short prefixes that are cheaper to predict.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a loss that the decoder can satisfy without the tokens.
You are building a video tokenizer for a world model. It turns each clip into a sequence of tokens that an autoregressive model predicts and a decoder renders back into pixels. The sequence runs coarse to fine, so any prefix can be decoded.
The road ahead
- 1 · The problem ~7 min A coarse-to-fine video tokenizer should pack a clip's meaning into its first tokens, but the current alignment loss can be met without the tokens.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Put the semantic signal on the tokens themselves: feed DINO features into the encoder and make heads rebuild them from each token prefix alone.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Encode with DINO features, train heads to rebuild them from each prefix, then predict short prefixes autoregressively and decode them to video.
- 6 · The experiment ~6 min You’ll guess: A 201M SemanTok AR model matches or beats a VideoFlexTok AR model how many times its size?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.