Every Weight Is a Path of Shared Discrete Decisions

Every Weight Is a Path of Shared Discrete Decisions. By Ammar Ali and one other, from the LEAD Lab at ITMO University. Published on the twenty-second of October. Its code repository is public and real: about thirty source files.

Making a neural network smaller or cheaper to run means choosing among separate techniques: pruning, quantization, low-rank factorization, codebooks, weight sharing. Today each compression method brings its own format and its own solver, so choosing among them is a matter of handling each one apart. Hardware has a parallel problem. A circuit whose weights are fixed at fabrication needs a dedicated constant multiplier for every weight. And discrete solvers, which might search these choices well, reach only narrow subproblems such as rounding. Three questions that touch the same weights are answered in three separate places.

TWIG, short for Tree-structured Weights with Inherited Gates, is a representation in which every weight of a neural network is the end of a path of small discrete decisions. A decision can be a sign, a shift by a power of two, a zero gate, or an index into a shared table. Weights that belong together share the upper part of their paths. Each decision is inherited by every weight below it in the tree, so it is stored once and applied once, however many weights it governs. The authors call this a simple change, and they draw three consequences from it.

First, any trained network can be written as a TWIG. When the decisions are powers of two, its weight layers become circuits made almost entirely of wires and adders, which suits chips whose weights are fixed at fabrication. Second, pruning, quantization, low-rank factorization, codebooks and weight sharing all become edits of one object. A single optimizer can then choose among all of them for a whole network, instead of each method being applied in its own format with its own solver.

Third, choosing the decisions is the ground-state problem of a higher-order Boltzmann machine, whose interactions follow the paths of the tree. That is precisely the problem class that annealers and quantum optimizers are built to solve. The representation also extends a direction weight formats have long been moving in. Group scales, shared exponents, microexponents and sparse weights all push formats toward shared decisions. TWIG takes that trend to its end, and every common format is a shallow TWIG at no extra cost.

The test uses five weight matrices of a large language model. The tree in question is a flat one: its signed power-of-two terms are chosen per weight for circuit cost, and it uses no group scales. It is compared against GPTQ at equal layer error, at three-bit and four-bit accuracy.

On typical matrices, the flat tree needs nine to fifteen percent fewer additions than GPTQ. On matrices with strong outliers the saving is larger: twenty to fifty-five percent fewer additions. Additions are what gets counted because the terms were chosen for circuit cost, and a weight layer built from powers of two is a circuit of wires and adders.

Fifty-five percent fewer additions, on the matrices with strong outliers, comes from a flat tree: one level of signed powers of two per weight, with no group scales and none of the deeper sharing the representation allows. The larger claim sits behind that figure. If every weight is a path of shared decisions, then compression formats, fixed-weight circuits and discrete solvers stop being three separate problems, and become three ways of working on one tree.