PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Cong Wei, Xuanchi Ren, Bryan Chu, Weiming Ren, Huan Ling, Jiahui Huang, Laura Leal-Taixé, Sanja Fidler and 3 others
NVIDIA · University of Waterloo
Published September 29, 2026 · on the feed October 1, 2026 · ▲ 20 · arXiv 2609.38597
The abstract
Unified Multimodal Models (UMMs) often rely on separate visual representations for understanding and generation, increasing visual context length and complicating integration with established vision-language pretraining pipelines. Recent advances in pixel-space modeling offer an encoder-free alternative, but extending this paradigm from images to videos is non-trivial: video understanding and generation adopt different temporal representations, leaving the design of a unified visual interface an open question. We present PixelUMM, an encoder-free model for unified image and video understanding and generation directly in pixel space. PixelUMM represents images as spatial patches and videos as spatiotemporal tubelets, connecting raw pixels to a shared multimodal backbone through single-layer linear projections. Its Mixture-of-Transformers architecture combines shared attention with task-specific parameters and extends clean-pixel prediction to video generation, jointly supporting autoregressive text prediction and pixel-space flow matching. Experiments show that PixelUMM achieves competitive performance across image and video understanding and generation tasks. We further conduct empirical studies of key design choices, including decoder design and spatial-temporal patch size, providing insights for future pixel-space unified multimodal models.
The Brief
2:21A short podcast summary of the paper. Written and voiced by AI.
Transcript
PixelUMM. Encoder-Free Unified Image and Video Understanding and Generation. By Cong Wei and ten others, from NVIDIA and the University of Waterloo. Published on the twenty-ninth of September. Its code repository is worked out: about sixty-five source files, with a readme and a licence.
Unified Multimodal Models are meant to understand images and generate them within one system. In practice they often keep two separate visual representations, one for understanding and one for generation. That has two costs: the visual context gets longer, and the model becomes harder to fit into established vision-language pretraining pipelines. Recent work on modeling directly in pixel space offers a way out, with no encoder at all. But taking that approach from images to videos is not straightforward, because video understanding and video generation use different temporal representations. What a unified visual interface should look like for video is still an open question.
PixelUMM is an encoder-free model that answers it, handling understanding and generation for both images and videos, directly in pixel space. The interface is deliberately thin. Images become spatial patches. Videos become spatiotemporal tubelets, patches that extend through time. Each is connected to a shared multimodal backbone by a single linear projection layer.
The backbone is a Mixture-of-Transformers architecture, combining shared attention with parameters specific to each task. It extends clean-pixel prediction to video generation, and it supports two objectives together: autoregressive prediction of text, and flow matching in pixel space.
Across image and video tasks, for understanding and for generation, PixelUMM achieves competitive performance; the abstract does not give figures. The authors also run empirical studies of key design choices, including how the decoder is designed and what patch size to use across space and time, and present them as guidance for future unified models working in pixel space.
Strip the encoders away and the question becomes how raw pixels should enter a shared model at all. PixelUMM's reply is about as minimal as it could be: patches for images, tubelets for video, one linear layer each, with one backbone carrying both understanding and generation. Its claim is competitiveness rather than dominance, and its design studies are offered to whoever builds the next model of this kind.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One model, two jobs, and two visual streams getting in the way. That is the wall.
You are building one model that both understands and generates images and videos. The usual design feeds it two visual streams: semantic features for understanding, compressed latents for generation.
The road ahead
- 1 · The problem ~7 min One model that both reads and makes images and videos usually sees every picture twice, which wastes context and breaks standard pretraining.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Feed raw pixels straight in as patches and tubelets, let one shared-attention backbone serve both tasks, and train text and pixel objectives together.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Cut pixels into patches or tubelets, project each with one linear layer, run all tokens through a routed shared backbone, and train both objectives together.
- 6 · The experiment ~6 min You’ll guess: With no pretrained vision encoder and no autoencoder, how did PixelUMM fare across image and video understanding and generation tasks?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.