FairRSFM: A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models

FairRSFM. A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models. By Md Aminur Hossain and four others, from the Space Applications Centre, ISRO, the Indian Institute of Technology Bombay, and one other institution. Published on the fifth of October. Its code repository is public but slight: seven source files.

Remote sensing foundation models, RSFMs, are commonly evaluated with aggregate metrics. An aggregate can look healthy while hiding systematic performance disparities between ecological regions, so a model that serves one kind of landscape well and another poorly still reports a single respectable number. For a model meant to observe the whole planet, the regions where it does worst are precisely the part an average is built to smooth over, and nothing in the usual evaluation asks for them separately.

FairRSFM is a biome-aware benchmark aimed at that blind spot: ecological group robustness. It takes georeferenced samples from fourteen terrestrial biome classes and maps them into six ecologically meaningful macro-groups. With those groups in place, performance can be read for each group as well as overall, and in particular for the worst group.

Models are evaluated under a unified frozen-backbone protocol, across four downstream datasets: m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type, and MMEarth20K with Dynamic World label maps. Three models are tested, Prithvi-EO-2.0, SatMAE and DOFA, each across three random seeds. Freezing the backbone means every model is judged on the representation it already has, so differences between biome groups are differences in what the foundation model itself learned, not in how it was later fine-tuned.

Across architectures and tasks, aggregate performance consistently masks disparities that depend on biome. Prithvi-EO-2.0 reaches ninety point nine eight percent overall macro-F1 on m-EuroSAT, but its mean worst-group score is only eighty-three point seven two percent. On m-SA-Crop-Type, an overall mIoU of twenty-seven point three percent drops to eighteen point four seven percent in the Xeric and Mineralogical group.

The benchmark also evaluates three mitigation baselines, meant to complement one another: Biome-Orthogonal Linear Probing, or BOLP, Dynamic Biome Reweighting, or DBR, and GroupDRO. How well they work depends on the model and the task. In one case, BOLP lifts the worst-group F1 score of Prithvi-EO-2.0 on m-BigEarthNet from forty-six point one two percent to fifty point two seven percent, without updating the backbone at all.

Code and datasets are public, and the authors offer FairRSFM as a reusable protocol for diagnosing and mitigating ecological robustness gaps. One pair of figures from it carries the point. Ninety-one percent overall and eighty-four in the worst group describe the same model on the same dataset, and an aggregate metric only ever shows the first. The mitigation results add a second, quieter point: no single fix closes the gap everywhere, so the diagnosis has to come first.