Tree-Ising: Repairing Language Models with Bit-Level Ising Edits

Tree-Ising. Repairing Language Models with Bit-Level Ising Edits. By Hadi Salloum and one other, from QDeep and ITMO University. Published on the tenth of November.

Weight-space programming, introduced in recent MIT work by Takehi and colleagues, lets a frontier LLM agent read a network's weights and write a repair. Their GPT-6 Astra agent corrected the reasoning loops of Qwen3.5-0.8B without gradients. The repair was not cheap in interaction. It took one hundred and fifty-eight agent turns, eighty-nine point seven million tokens, and fifty forward-pass requests. The question this work asks is whether the same kind of repair needs a reading agent at all, or whether it can be posed directly as a problem over the weights and solved.

Tree-Ising takes the same repair and recasts it with no agent in the loop. It becomes a sparse discrete optimization problem over the model's own BF16 weights. Forward statistics define the objective, which pulls three ways at once: weaken a loop direction, strengthen a concluding direction, and preserve the layer's output on productive tokens. In plain terms, the model should be nudged away from going round in circles, toward finishing its answer, and should otherwise behave as it did before on the tokens where it was already doing useful work.

The edits are kept on a short leash. Within a state-tree representation, where lossless BF16 storage acts as a special case, the only admissible edits are sign flips and exponent shifts, and those must preserve all shared group exponents and mantissas. The objective then decomposes exactly into independent QUBO problems, which are Ising problems, one per output row. These are solved on a GPU, using either simulated annealing or quantum-inspired simulated bifurcation. Because the decomposition is exact, handling the rows one at a time does not change the problem being solved.

Start with the agent's own two layers. There, Tree-Ising reaches fifty-four point six percent on MATH-500, against fifty-one point seven percent for the agent's edit under the same evaluation protocol. The base model scores seven point nine percent, as reported. The simulated bifurcation solver gives a statistically indistinguishable repair, at fifty-five point one percent. Both solvers outperform the agent edit across all five math benchmarks, with statistically significant improvements on MATH-500 and GSM8K after correcting for multiple comparisons.

The repair is also lighter. It modifies thirteen to seventeen percent fewer weights than the agent's, and cuts BF16 bit flips by a factor of two point two to two point three. It needs only four forward-pass requests and no LLM tokens at all, and the annealing recipe costs under a dollar in GPU time, about nine tenths of one.

A larger model tests whether the recipe carries over. On Qwen3.5-2B, with the layers and the dose selected using only the base model, the same recipe lifts greedy GSM8K accuracy from twenty-one point six percent to sixty-nine point four percent. As a proof of concept, an eight-variable repair subproblem is also executed using QAOA on an IBM Heron processor.

Set the two repairs next to each other. One reads weights through a frontier agent over many turns and millions of tokens. The other writes the same kind of fix as a set of small binary choices about signs and exponents, solves them row by row, and does at least as well while touching fewer bits. Framing the edit as an Ising problem is also what lets a piece of it run on quantum hardware.