Spectral-Spatial Interpretation

Spectral-Spatial Interpretation. By Haotian Ma and two others, from the University of Wisconsin-Madison. Published on the fifteenth of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

Explanations of image models after training usually take one of two forms. A spatial heatmap shows where the model looks. A set of global channel scores says which channels matter overall. For ordinary photographs that split may be tolerable. For multispectral data it is limiting, because the relevance of a channel is often localized in space and physically meaningful there. A band can matter in one part of the image and not in another, and neither a single heatmap nor a single ranking of channels can say so.

Spectral-Spatial Interpretation, or SSI, attributes over the full grid. Its players are pairs of a position and a channel, written g and c. The output is a complete spectral-spatial attribution matrix, giving a score to every channel at every position, rather than collapsing everything into one heatmap or one global channel ranking.

The construction works one position at a time. For each spatial position, SSI holds the observed surrounding context fixed. It then defines a conditional game over the channels at that position, and applies the Shapley value to this local game. Each position therefore gets its own fair division of credit among its channels, computed with the rest of the image left as it was observed.

That gives SSI a clear local interpretation. It satisfies Shapley fairness within each position. And if its channel attributions at a position are added up, they exactly recover the position-level contextual contrast: the effect of replacing that position with the baseline. The authors also separate SSI from context-averaged Owen-style hierarchical accounting. That approach uses the same grouping of positions and channels, but averages over external spatial coalitions instead of fixing the observed context.

On HLS Burn Scar, SSI outperforms an MC-Owen estimator across the retained attribution metrics, while using only about six percent of its evaluation budget.

The broader comparison spans CUB-200, EuroSAT and HLS Burn Scar, under two baselines: a per-channel mean, and zero. Against dense, flat and rank-1 baselines, SSI achieves the best joint deletion and insertion AUC. It yields the strongest spectral hit rates and channel deletion results, and it leads on spatial localization on both annotated datasets. Code is available.

The method changes the unit of explanation. Instead of asking where in the image, or which channel in general, it asks which channel, here. Fixing the observed context around each position is what makes that question local and exactly accountable, and it is also what lets SSI do the job on a fraction of the evaluation budget of the averaged alternative.