RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

RGBD20K. A Large-Scale Benchmark for RGB-D Semantic Segmentation. By Shaohua Dong and eight others, from the University of North Texas. Published on the twenty-fourth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

RGBD20K is a new dataset for RGB-D semantic segmentation: the task of labelling every pixel of a scene when each colour image comes paired with depth. The authors built it with a specific aim. They want the field to develop segmentation that is more robust and more general, and they argue that the route there runs through two things the data has to supply: many more categories, and annotations that can be trusted.

Three properties define it. The first is an expanded semantic space. RGBD20K covers one hundred and sixty fine-grained categories. For comparison, the abstract names two popular existing benchmarks: NYUv2, with forty classes, and SUN RGB-D, with thirty-seven. The second property is scale. The dataset offers twenty thousand RGB-D image pairs, which the authors describe as a substantially larger training resource than current benchmarks provide. The third is annotation quality, and that is where the construction story begins.

The labels were not simply collected and shipped. The authors performed a rigorous re-evaluation and correction of existing labels, aimed at long-standing annotation noise, and describe the result as a clean and reliable ground-truth foundation. Alongside the data they propose a method of their own, called score-purified fusion, or SPF. The abstract says it achieves state-of-the-art performance across all evaluated benchmarks, and presents that as evidence of how well high-quality multimodal information can be used for this task. It does not describe how the fusion works, so neither will this episode. The dataset is released publicly through the authors' repository.

The natural users are people training RGB-D segmentation models who want more categories than the older benchmarks name. The authors expect the richer semantic coverage to promote models that generalise better, and the larger scale to support more powerful deep models. Anyone who has tuned against noisy ground truth in the older benchmarks is the other audience, since the correction pass is aimed squarely at them.

RGBD20K offers the field more of what it trains on, in three senses at once. Four times the categories of NYUv2, a larger pool of paired images, and labels that have been checked and fixed rather than inherited. If segmentation models have been limited by what their data could name and how cleanly it named it, this is the resource built to test that.