Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations
Junru Hu, Cheng Tang, Wei Li, Hongze Zhu, Chun-Mei Feng, Qi Zhang, Yirong Chen, Ming Hu
Shanghai Artificial Intelligence Laboratory · Tongji University · Fudan University
Published November 6, 2026 · on the feed November 9, 2026 · ▲ 18 · arXiv 2611.09382
The abstract
Weight-Space Programming (WSP) enables reasoning agents to directly inspect and modify neural network parameters, offering an alternative to gradient-based model adaptation. However, its capabilities, cross-task generalization, and practical limitations remain insufficiently understood. We systematically investigate WSP through 16 editing trajectories across seven benchmarks and two model scales (Qwen3.5-0.8B and Qwen3.5-9B). Our findings reveal three key insights. First, WSP can achieve substantial performance gains through highly localized interventions. On full held-out evaluations, accuracy improves by up to 29.45, 4.42, 36.72, and 17.11 percentage points on MATH-500, GPQA Diamond, IFEval, and SciBench, respectively. Notably, modifying only 1,023 parameters yields a 29.45-point improvement. Second, WSP exhibits promising but uneven cross-task generalization. A Math-edited checkpoint improves GPQA Diamond, IFEval, and SciBench by 3.03, 12.18, and 20.10 points, respectively, without further editing, while leaving SciCode performance unchanged. Third, effective editing remains challenging and inconsistent. Outcomes vary substantially across trajectories, and six of the 16 trajectories retain the original weights when candidate improvements cannot be reliably validated. Together, these results highlight WSP's potential for highly localized model adaptation while revealing important challenges in intervention discovery, reliability, and cross-task transfer. Homepage: https://wsp-scope.github.io/ GitHub: https://github.com/WSP-Scope/WSP-Scope-Project
The Brief
2:17A short podcast summary of the paper. Written and voiced by AI.
Transcript
Exploring Weight-Space Programming. Capabilities, Generalization, and Limitations. By Junru Hu and seven others, from the Shanghai Artificial Intelligence Laboratory, Tongji University, and one other institution. Published on the sixth of November.
Weight-Space Programming, or WSP, lets a reasoning agent look directly at a neural network's parameters and change them. It is an alternative to adapting a model by gradients. What it can do, how far its edits generalize across tasks, and where it breaks down have not been well understood, and this study sets out to map those three things. Its answer comes in three parts: WSP can deliver large gains from very localized edits, its transfer to other tasks is promising but uneven, and finding good edits reliably is still hard.
The investigation runs sixteen editing trajectories over seven benchmarks and two model scales, Qwen3.5-0.8B and Qwen3.5-9B. On full held-out evaluations, accuracy rises by as much as twenty-nine point four five percentage points on MATH-500, four point four two on GPQA Diamond, thirty-six point seven two on IFEval, and seventeen point one one on SciBench.
How localized can an edit be? Changing only one thousand and twenty-three parameters produces that twenty-nine point four five point improvement.
Transfer was tested with a checkpoint edited for math and then left alone. Without any further editing, it improves GPQA Diamond by three point zero three points, IFEval by twelve point one eight, and SciBench by twenty point one zero. SciCode, by contrast, does not move at all.
The limits are stated as plainly as the gains. Outcomes vary substantially from one trajectory to the next. In six of the sixteen trajectories, the process keeps the original weights, because candidate improvements could not be reliably validated. The results come from two Qwen3.5 model sizes, and the authors frame the open problems as three: discovering the right intervention, making editing reliable, and getting edits to transfer across tasks.
The picture is of a tool with a high ceiling and an unsteady floor. A tiny, targeted change to the weights, touching only a sliver of the parameters, can move a math benchmark by double digits and sometimes carry over to other tasks. Getting there consistently is the part that remains unsolved, which is why the authors present WSP as promising for highly localized adaptation and not yet as a dependable one.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free