Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Marigold V2. Revisiting Diffusion Transformers for Monocular Depth Estimation. By Igor Pavlovic and eight others, from EPFL, the Huawei Bayer Lab, and one other institution. Published on the eighth of September. Its code repository is worked out: about one hundred source files, with a readme and a licence.

Monocular depth estimation, recovering depth from a single image, is everywhere and deeply ill-posed. It feeds scene reconstruction, computational photography and robotics, among other applications. The field is mature, and yet recent models still struggle in two ways: generalising to out-of- distribution inputs, and producing depth maps that are sharp and detailed.

Marigold V2 revisits Marigold, a set of techniques for turning modern image generation and editing models into state-of-the-art monocular depth estimators. This time the models being repurposed are powered by the diffusion transformer, or DiT, architecture. The recipes aim for single-step inference from pretrained flow-matching models that normally run in many steps, with quantisation where it is needed, so the model keeps its capacity while staying cheap to run. A generator that takes many steps to produce an image is made to produce a depth map in one, keeping what it learned while shedding most of the cost.

The authors analyse the artifacts that naive training produces, and identify two effective remedies. The first aligns the model's internal representations with semantic features extracted from the ground truth. The second is a fine- tuning protocol in two stages, built around a new loss based on Sinkhorn. One remedy acts on what the model represents internally, tying it to the meaning of the scene. The other acts on how the model is trained, in a staged schedule with a loss designed for it.

The resulting depth maps are crisper and cleaner, and they generalise well out of distribution. On KITTI and ETH3D, AbsRel improves by sixteen to twenty-six percent over the previous best. The comparison in both cases is with the previous best result. Qualitatively, the model resolves fur, foliage, and edges as thin as a hair, details that have eluded prior models. And beyond depth, Marigold V2 reaches state-of-the-art results on other dense regression tasks, such as estimating surface normals and decomposing an image into its intrinsic components.

The two weaknesses named at the start, generalisation and detail, are both addressed by the same move: take a large generative model that already understands images, and teach it to output geometry in one step. The remedies for naive training are what let that capacity show up as sharp edges rather than artifacts, and the recipe carries over to other dense tasks.