GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats

GaussianRenderFormer. Transformer-based Rendering of 3D Gaussian Splats. By Shahaf Valencia Levy and one other, from the Hebrew University of Jerusalem. Published on the third of November.

The question is a plain one. Can a neural network learn to render 3D Gaussian splats by itself, with no rasterizer anywhere in the renderer? GaussianRenderFormer is the authors' attempt at it: a transformer that takes a set of Gaussians and a camera, and outputs the image directly. Each Gaussian becomes one token in an encoder that does not depend on the view. A second part, a decoder that does depend on the view, turns small patches of camera rays into pixels by attending to those tokens. The answer they report is a qualified yes. The network renders, but it does not yet reach the rasterizer.

Training uses twenty-six thousand eight hundred and twenty Objaverse objects, each pruned to twenty thousand Gaussians. On three hundred unseen objects, measured against the full, unpruned splats, GaussianRenderFormer reaches thirty-eight point zero eight decibels of PSNR. Rasterizing the same pruned input reaches forty-four point zero two.

What the paper is careful about is where that gap comes from. It is not a failure to generalize. The diagnostics are consistent with a bottleneck in the view decoder. One of them makes the point: telling the decoder the projected footprint of each Gaussian, something it could in principle work out for itself, adds one point four decibels. The authors also say plainly that under-training is not ruled out, so the bottleneck reading is offered as consistent with the evidence, not as settled.

A learned renderer for splats works, and it falls short of a rasterizer by about six decibels on this test. Whether a bottlenecked decoder or too little training holds it back is the question left open.