GTR: Gated Token Recurrence for Efficient Dense Prediction

GTR. Gated Token Recurrence for Efficient Dense Prediction. By Zhe Feng and nine others, from Didi International Business Group, Intellindust AI Lab, and three other institutions. Published on the twenty-third of September. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.

Vision backbones built on self-attention do well on dense prediction. The trouble is cost. Global softmax attention scales quadratically, so as image resolution rises, efficiency falls, and high-resolution dense prediction is exactly where that cost bites.

Gated Token Recurrence, or GTR, removes softmax attention altogether. It is a recurrent vision backbone combining three ingredients: gated linear attention, spatial scan directions that alternate, and SwiGLU blocks that are spatially enhanced. Tokens are mixed by recurrence instead of every token attending to every other.

The training recipe is minimal. GTR is distilled from a DINOv3 teacher specialized for detection, and the only signal is alignment of final-layer patch tokens, through a linear projection and a squared ell two loss. There is no masked-token prediction and no supervision of intermediate layers. On the systems side, the authors write a specialized chunkwise CUDA operator. In an isolated kernel benchmark on an RTX 4090, it runs four times faster than FLA version zero point five, at one point six thousand tokens.

With detector pre-training on Objects365, GTR-L reaches fifty-eight point nine box AP on COCO val2017. Its median batch-one latency is one point nine zero eight milliseconds, under compiled FP16 execution on an RTX 4090. The same backbone transfers to instance segmentation, pose estimation, oriented detection, semantic segmentation and monocular depth estimation.

Deployed with TensorRT on DRIVE AGX Thor, the evaluated models run at a median batch-one latency between two point two eight two and eight point seven six nine milliseconds.

The conclusion the authors draw is specific. Recurrent token mixing can be an efficient alternative to global softmax attention, both for high-resolution dense prediction and for deployment at the edge. GTR is the case they make for it: a softmax-free backbone, distilled from one teacher's final layer, that carries over to six dense tasks and stays in the low milliseconds on an RTX 4090 and on DRIVE AGX Thor alike.