Semantic Concept Steering Breaks the Explanation Drift Loop in Continual Learning

Semantic Concept Steering Breaks the Explanation Drift Loop in Continual Learning. By Yehonatan Elisha and two others, from Tel Aviv University and the Open University. Published on the twenty-fourth of October. No code repository is linked to it.

Catastrophic forgetting in continual learning is usually measured as lost accuracy. This work points to a quieter form of it, explanation drift. A model can keep its predictive accuracy while silently shifting its attention away from diagnostic features and onto spurious correlations. The authors then identify a structural flaw in the methods meant to guard against this. In explanation-aware continual learning, pixel-level saliency regularization takes its supervision targets from the model's own frozen saliency maps. That makes the loop self-referential. Whatever spurious attention the model already had becomes the target, and the regularizer actively reinforces it from one task to the next.

The proposed fix is Concept-Consistent Continual Learning, written C cubed L. It breaks the loop by anchoring regularization to something outside the model: class-consistent semantic concepts, things like a curved beak or a striped pattern, discovered offline and independent of the model's state. That buys two properties. The targets cannot drift when the model drifts. And the concepts are consistent across every instance of a class, rather than varying image by image. Where the old targets came from inside the model and moved whenever the model moved, these targets come from outside it and stay fixed. The supervision no longer depends on the very thing it is supposed to be correcting.

The authors also identify a crowding-out effect. Their concept-based objective is combined with approximate LRP relevance conservation, and under that conservation, concept regions and background regions compete for the relevance of the same prediction. Push relevance up inside the concept regions and the background has less to claim. Spurious background correlations are suppressed implicitly, without any explicit penalty on them. Nor does the method need human-annotated concept sets: concept discovery and spatial grounding are fully automated.

Evaluation spans five benchmarks and more than twenty baselines. The reported result is state-of-the-art performance on accuracy, on forgetting, on explanation quality, on spurious correlation mitigation, and on out-of-distribution robustness. That list matters for the argument. The claim is not only that the model forgets less, but that what it attends to stays on the diagnostic features rather than sliding toward the background, which is the drift the work set out to stop.

The diagnosis is the part to carry away. A regularizer that copies the model's own past explanations cannot correct them, only preserve them, mistakes included. Moving the reference point to concepts fixed in advance is what lets the explanation stay put while the tasks change.