VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

VisionHOPE. Visual Backbones as Self-Modifying Learning Systems. By Siran Peng and ten others, from Mininglamp Technology, the Institute of Automation, Chinese Academy of Sciences, and two other institutions. Published on the twenty-seventh of September. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.

Visual backbones have moved through a sequence of designs. Convolutional Neural Networks aggregate locally. Vision Transformers interact globally. State-Space Models make their state transitions depend on the input. And Test-Time Training layers adapt an inner learner while an image is being processed. Along that path, computation has grown steadily more adaptive to each input. What has not changed much is who sets the rules for that adaptation. Those rules are still largely prescribed by the trained backbone. A model may update some inner state as it reads an image, but the way it updates was fixed when training ended. That is the next step the progression has not yet taken.

VisionHOPE is presented as the first generic visual backbone formulated as a self-modifying learning system. Within a single image, what the model remembers and how it learns evolve together. Memory and learning rule are not separate things set at different times. Both change as the image is read, and each shapes the other. It builds on the self-referential construction of Nested Learning, abbreviated NL.

The co-evolution is carried by five coupled memories. Between them they store content, generate key and value representations, and govern the learning rate and how much is retained. All five evolve jointly as visual context builds up along each scan. Notice that two of the five are about learning itself: the rate at which things change, and how much is kept. Those are exactly the quantities a conventional backbone would fix in advance.

Applying the unconstrained self-referential update directly to a visual backbone turns out to be unstable, so the authors derive a stability-matched control scheme for the step size. It combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and they prove that the resulting memory dynamics are non-expansive along each scan. Images are two-dimensional, so a further adaptation is needed. The chunk formulation from NL is aligned with the rows and columns of the image, across four directional scans. The sequence-style machinery of the original construction is thereby fitted to the grid of an image, with each scan direction reading the feature map in its own order.

VisionHOPE reports competitive results on ImageNet-1K, on COCO, and on ADE20K. On the strength of those results the authors argue that self-modifying learning systems are a practical foundation for general-purpose visual backbones. The code is available.

Each earlier backbone made the computation more responsive to its input while keeping the learning rule fixed. VisionHOPE lets the learning rule move too, inside one image, and then spends real effort making sure that movement stays bounded. The stability proof is what turns a self-referential idea into something that can sit underneath ordinary vision tasks.