RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives
Chong Zeng, Yue Dong, Pieter Peers, Lvmin Zhang, Maneesh Agrawala
Stanford University · Microsoft Research · College of William & Mary
Published September 4, 2026 · on the feed September 9, 2026 · ▲ 5 · arXiv 2609.05738
The abstract
We present 'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism.
The Brief
2:13A short podcast summary of the paper. Written and voiced by AI.
Transcript
RenderFormer-V2. Neural Rendering with Heterogeneous Scene Primitives. By Chong Zeng and four others, from Stanford University, Microsoft Research, and one other institution. Published on the fourth of September. Its code repository is worked out: about one hundred and sixty source files, with a readme and a licence.
Rendering an image means resolving global light transport, and some effects are notoriously demanding: caustics, volumetric scattering, environment lighting, textured and displaced surfaces, and materials outside what a system was built for. A learned renderer that handles all of these usually needs per- scene training or specialised code for particular effects. RenderFormer-V2 aims to handle that diversity with neither, as a unified learned model that complements modern physics-based rendering systems rather than replacing them.
RenderFormer-V2 is a transformer-based neural rendering model that treats global light transport as a sequence-to-sequence transformation. Following its predecessor, it works in two stages. A view-independent stage resolves how light moves from primitive to primitive within the scene. A view-dependent stage then transforms that internal neural scene representation into image pixels.
What changes from RenderFormer sits mostly in the view-independent stage. There the model combines windowed attention with a rendering-informed attention sink, a new mechanism meant to improve scalability while keeping render accuracy. Versatility is extended in two further ways. The model supports heterogeneous scene primitives, including environment maps and participating media, not only surfaces. And it encodes materials independently of the underlying surface reflectance model, capturing material appearance through a new neural embedding.
The authors demonstrate the model's versatility across a variety of scenes, and they perform an extensive ablation of the improved attention mechanism. The scenes are chosen to show the range of effects and primitives the model covers, and the ablation isolates the contribution of the combined windowed attention and attention sink.
The direction of the update is breadth. Its predecessor established that light transport can be learned as a sequence transformation. This version asks that one model take in more kinds of scene content, from media to environment maps to arbitrary materials, and scale further, without training for each scene or writing code for each effect.
Paper Quest · 7 chapters · about 37 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A renderer that learned light from data, held back by its own attention. Start there.
You have a learned renderer: a transformer reads a scene's triangles as tokens and outputs an image with full global illumination, no per-scene training needed.
The road ahead
- 1 · The problem ~6 min A learned renderer that lets every triangle token attend to every other one gets too costly and loses focus once scenes grow large.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Each token attends to a window of nearby tokens plus a few shared sink tokens holding the lights and summaries, so cost stays bounded and focused.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Scene pieces become tokens, a sparse-attention stage works out light between them, and a second stage turns those tokens into pixels for the camera.
- 6 · Field test ~3 min
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.