Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Frozen Models, Evolving Expertise. Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI. By Yexiao He and eight others, from the University of Maryland, College Park and NVIDIA. Published on the sixth of October.

Large language models and vision-language models are usually frozen once they are deployed, which means they learn nothing from the cases they solve. In medicine that is a particular concern, because new clinical evidence, updated guidelines and new therapies can change what established practice is. The existing remedies each fall short somewhere. Fine-tuning can update a model, but it needs access to the weights and more training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual detail by saving experience only as text.

The authors present a model-agnostic framework that lets frozen language and vision-language models learn from deployment experience without touching the model itself. The learning lives outside the model, in three forms of external expertise, and a validation strategy decides which updates to that expertise are kept. Each form is aimed at one of the shortcomings just listed. No weights are needed, so closed models qualify. Facts are kept only when supported, so the knowledge is meant to be reliable. And images are stored as images, so visual detail is not flattened into words.

The first form is a Skill, which guides reasoning and tool use. The second is a Knowledge Memory, which stores reliable facts, supported either by earlier cases or by trusted external evidence. The third is a Multimodal Knowledge Base. It keeps visual examples rather than text alone, and it guides the model to relate each case it retrieves to the image currently in front of it. On validation, the framework does not lean on a fixed validation set. An update is kept only if it helps on new cases without degrading performance on earlier ones. That rule is the answer to overfitting a fixed set: the test an update must pass keeps moving with deployment, and it guards against forgetting at the same time, since nothing is admitted that breaks what already worked.

Testing covers six benchmarks spanning clinical diagnosis, clinical workflows, medical reasoning, and visual reasoning both medical and non-medical, with four base models drawn from open-weight and closed-source systems. During online deployment, the framework improves performance on medical tasks by up to thirty-four point two percent over the base model. It generalizes to unseen cases, it transfers to other models without further optimization, and it also works outside medicine. That transfer result follows from the design: because the expertise lives outside any one model, what was learned with one base model can be handed to another.

What the framework separates is the model from what the model has learned on the job. The weights stay frozen and can even be closed; the expertise accumulates beside them, in a form that includes images, and it is only allowed to grow when growing does not cost earlier cases anything.