PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion

PixelDense. Dense Prediction as Representation Alignment for Pixel Diffusion. By Lehan Yang and eleven others, from the University of Virginia, Michigan State University, and two other institutions. Published on the thirtieth of September. Its code repository is public but slight: one source file.

Representation alignment, known as REPA, speeds up the training of diffusion transformers by aligning them to the representations of another model, a teacher. Its alignment targets have almost always been semantic encoders, such as DINOv2 and CLIP. Recent analysis points somewhere else. It suggests that spatial structure, not global semantics, is what actually carries the alignment effect. Yet there is a family of foundation models trained to predict exactly that structure, the dense-prediction models, and they have been overlooked as alignment targets.

Tried in pixel-space diffusion, they hold up well. SAM2, Depth Anything v2 and Metric3D v2 each outperform the GenEval baseline that aligns to DINOv2 alone, and the two geometric teachers lead the segmentation teacher. Combining them is where things go wrong. A flat sum of all four teachers lands below the best single geometric teacher on its own. The reason given is a conflict: semantic gradients and geometric gradients compete for one denoiser projection, and adding more teachers into that one channel makes the result worse rather than better.

PixelDense resolves that competition by giving each kind of signal its own route into the model. DINOv2 and SAM2 go through a semantic projection stream. Depth Anything v2 and Metric3D v2 go through a separate geometric projection stream. The two kinds of teacher no longer have to share.

A weight-space orthogonality penalty then keeps the two streams in disjoint subspaces, so that they stay apart during training. All four teachers are frozen throughout, and all four are dropped at inference, so they cost nothing at sampling time. One single recipe is applied, unchanged, to two different models, PixelGen and DeCo.

PixelDense improves GenEval, DPG-Bench and HPS v2.1. On PixelGen-XXL, GenEval Overall rises from zero point seven nine two seven to zero point eight zero nine three. PixelDense also beats every single-teacher variant, and every unfactored multi-teacher variant, which is the comparison that tests the split into streams. Trained from random initialization, it reaches the baseline's peak GenEval one point two three times faster.

Independent probes add a second view, looking at what the model can reconstruct. In partial-noise reconstruction, probes for panoptic, depth and surface-normal prediction show up to fifty-three point one percent gain in PQ, and a thirty-six percent reduction in depth AbsRel, at tau of zero point five across COCO and Flickr30K. Editing tells a similar story. In SDEdit editing on PIE-Bench, PixelDense keeps more of the source background and layout at every edit strength, raising background PSNR by up to two point two decibels.

Two observations drive the whole design. Geometry is a strong alignment target on its own. Geometry mixed naively with semantics does worse than geometry alone. Put those side by side and the remedy follows: keep the two kinds of signal in separate subspaces, and all four teachers can contribute at once instead of pulling against one another.