Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition

Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition. By Tahar Chettaoui and four others, from the Fraunhofer Institute for Computer Graphics Research and TU Darmstadt. Published on the thirtieth of October.

Multi-head attention in Vision Transformers lets each head form its own attention pattern. Without explicit regularization, though, heads may learn redundant patterns instead of complementary roles. Correlation-based penalties exist to push heads apart, but they regularize every training sample equally. In face recognition, samples are not equal. Hard samples, the ones whose embeddings sit close to competing identities, are critical for both optimization and generalization, and a uniform penalty treats them like everything else.

ViT-HarDis makes head disentanglement conditional on how hard a sample is. Difficulty is read from cosine similarity to the centers of negative classes. A sample that looks a lot like some other identity receives stronger head decorrelation, so specialization is enforced exactly where identity is most ambiguous. An easy sample, far from every other identity, is left mostly alone. The penalty, in other words, scales with the very quantity that makes a face hard to recognize.

Two further choices shape the method. Disentanglement is applied only to the final transformer layers, and the authors' analysis shows this outperforms regularizing the entire network. And the change lives entirely in training: there are no architectural modifications and no additional inference cost. The same analysis shows the two intended effects together. Hard samples become less similar to competing class centers, and attention across heads becomes less similar.

Experiments show improvements over both the baseline and uniform disentanglement on most benchmarks. Trained on WebFace4M, ViT-HarDis reaches state-of-the-art results among ViT-based methods on IJB-B, on IJB-C, and on TinyFace Rank-1. Trained on MS1MV3 instead, it achieves the best IJB-B and TinyFace results. Two training sets, and leading results on each, with the gains coming from a training-time change alone. The abstract frames the benchmark wins with care: improvements on most benchmarks rather than all, and state of the art among ViT-based methods specifically.

A regularizer already in use is not replaced here but aimed. Where correlation penalties once spread their pressure evenly, this one concentrates it on the samples that sit nearest the wrong identity, which is where redundant heads would cost a face recognizer the most.