Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

Neural Spectral Capacity. Measuring and Designing Architectures from Network Specification Alone. By Chenyu Zhu and two others, from the City University of Hong Kong. Published on the nineteenth of September. Its code repository is public and real: about thirty source files.

Designing a Transformer and compressing one come down to the same decision: how to allocate capacity under a budget. The numbers usually consulted for that decision are the parameter count and the FLOPs count. They capture size, and they capture compute, but they say nothing about architectural structure. Take two architectures with identical parameter budgets that allocate them differently, across depth and width, across heads, or across the feed-forward layers. Both receive the same score. They do not behave the same. A scalar that cannot tell them apart cannot guide the choice between them.

Neural Spectral Capacity, or NSC, is a closed-form scalar grounded in the singular-value spectrum of each weight matrix. The step that makes it practical is statistical. Under standard random initialization, the Marchenko-Pastur law describes that spectrum, and so NSC can be computed from the architectural specification alone. No model is instantiated. No data is needed, and no gradients.

Because NSC is additive layer by layer, the design question becomes one that dynamic programming can answer exactly. That solver, NSC-DP, returns the architecture that globally maximizes NSC under given resource constraints, and it does so in seconds on a CPU. The authors are explicit about what that buys. Black-box search over existing training-free proxies cannot guarantee a global optimum. An exact solver over an additive score can.

Ranking comes first. Across seven Transformer and CNN families, NSC ranks architectures better than parameter count, better than FLOPs, and better than representative training-free proxies. The hardest test is on FlexiBERT, on pairs of models whose parameter counts differ by less than ten percent. There NSC reaches a tau of zero point five zero five, while parameter count collapses to zero point zero eight two.

Then design, and compression. NSC-DP finds a Transformer-XL architecture on WikiText one hundred and three that beats the human-designed baseline, and the search takes two seconds. On compression, it prunes LLaMA seven billion down to what the authors report as the best five point seven billion parameter model across eight commonsense reasoning tasks. It needs no calibration data, and it runs about five thousand nine hundred times faster than the strongest training-free proxy baseline.

Underneath all three results is a single move. Once the Marchenko-Pastur law fixes what the spectrum of an untrained layer looks like, a structural measure of capacity becomes arithmetic on the specification sheet. Arithmetic that adds up across layers can then be optimized exactly, which turns choosing an architecture from a search into a calculation.