SolarRenderer: Few-Step Generative Reference Rendering

SolarRenderer. Few-Step Generative Reference Rendering. By Guian Fang and eight others, from the National University of Singapore, LIGHTSPEED, and three other institutions. Published on the twentieth of October. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

Game engines and coding agents can already lay out a scene, its motion and its camera. What they produce is coarse structure. Turning that into realistic video needs a renderer that obeys the structure and also fills in the appearance the structure leaves unspecified. The authors find three shortcomings in existing renderers. Each is tied to the representation of one particular engine. They need tens of denoising steps, or else a costly distillation of their own. And they learn from conditions that are not aligned with the videos they are meant to produce.

SolarRenderer is a structure-conditioned video renderer that leaves its large model untouched. It adapts a frozen MiniMax-H3 backbone of thirty-three billion parameters using two lightweight LoRAs, small add-on modules trained while the backbone itself stays fixed. One LoRA handles control, the other handles speed, and between them they take on the shortcomings just listed.

The first answers the problem of being tied to one engine. Structural videos enter the backbone's native reference pathway, placed at the positions of the frames they describe. They carry no type identifier, and are described only in plain text. Because nothing marks what kind of structure is coming in, a single control LoRA learns to follow arbitrary structural representations.

The second answers the step count. Starting from a pretrained four-step LoRA, structure-guided distribution matching renders in four steps. That is twelve point three times faster than sampling with fifty steps, while the control the model learned is preserved. A block-causal variant extends the same design to streaming, and renders seven hundred sixty-eight p video in real time.

For training data, the authors build a frame-aligned corpus of one hundred thirty-six thousand clips, drawn from game engines, simulators and YouTube videos. Frame alignment speaks to the third shortcoming, conditions that do not line up with their targets.

Evaluation uses an in-distribution test set and a generalisation benchmark covering sixteen structural representations. SolarRenderer achieves state of the art results on both. It also generalises to eleven representations it has not seen, among them pose skeletons, drawings and point clouds. Those unseen cases bear directly on the first design choice. A renderer told about its input only in plain text, with no type identifier, is being asked to cope with whatever structure arrives, and here it does so for kinds of structure absent from its training.

One renderer, few steps, any structure: each of the three complaints about existing renderers gets its own fix, and the backbone stays frozen throughout.