Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them

Broken on Arrival. Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them. By Aditi Patodiya, working independently. Published on the twenty-fifth of October.

Developers routinely download quantized large language models from public registries and run them on their own machines. None of the registries examined here documented any functional testing of the files they publish. The claim is that some of those files simply do not work, that they can be bit-for-bit faithful builds and still not work, and that a cheap screen catches them before anyone else has to find out.

The audit covered all three hundred and five eligible code-capable GGUF tags, the llama.cpp format, across fifteen model lines in the official Ollama library, along with twenty-two popular Hugging Face community files. A fifteen-task code screen picked out suspects, and those were retested on all one hundred and sixty-four HumanEval+ tasks. Seven official tags, two point three percent of the total, solved at most six of them. Every one was a low-bit quantization of a model with at most four billion parameters.

Five suspect files were then rebuilt from the original weights. All five registry files matched the uncalibrated rebuilds tensor for tensor, so nothing had been corrupted. Exact reproducibility did not imply function. Holding the weights, the toolchain and the tensor mix fixed, adding an importance matrix lifted three Qwen2.5-Coder-3B builds from zero solved tasks to between seventy-six and one hundred and thirteen.

The failure is not a property of the recipe by itself. The same uncalibrated recipe left the smaller Qwen2.5-Coder-1.5B functional, so the recipe alone does not predict which builds break. In the other direction, none of sixty-six sampled tags that passed the screen turned out to be non-functional on retesting. The failures that were found sit in one corner: low-bit quantizations of small models, all on code tasks.

The author releases the audit data and a tool called quantcheck, and recommends that low-bit tags be screened before publication, with failures either calibrated or withheld. A file can be a perfect copy of what its recipe produces and still arrive broken, which is why the check proposed here tests what the model does rather than what the file contains.

Broken on Arrival. Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them. By Aditi Patodiya, working independently. Published on the twenty-fifth of October. This is the long version: the whole paper rather than the abstract.

Running a language model locally now usually means pulling a quantized file from a public registry: a conversion of the model its authors evaluated, squeezed to between two and eight bits per weight. Neither the registry pages nor the file metadata the author examined record a functional test before anyone downloads it. Ask one of these files for a function that returns the length of a string, and you may get a fluent preamble, a signature, a docstring, and no body. Another streams the word program, over and over, until it hits the token cap. These files do not crash and do not refuse. They run at normal speed and produce nothing usable. The question is how often that happens across a whole registry, and why.

Two ideas come first. The format is GGUF, the single-file format of llama.cpp, the engine Ollama is built on. Its k-quants store weights in blocks with per-block scales, and a level name like q three K M names a fixed mix of tensor types rather than a single bit width. The second idea is calibration. An importance matrix holds activation statistics gathered by running some calibration text through the model, and llama.cpp uses it to weight the rounding error when it chooses block scales. Without one, the file is quantized from the weights alone. The toolchain refuses to make its very lowest-bit types without an importance matrix, but exempts the two-bit and three-bit k-quant levels by design. Keep two properties apart: whether a file is exactly what its recipe produces, and whether it works.

The census takes every eligible tag in fifteen model lines of the official Ollama library: instruct, code-capable, with an explicit quantization suffix, and at most eight gigabytes. That is three hundred and five tags, plus twenty-two popular community files from Hugging Face. The screen is fifteen coding tasks. It was tuned on twenty-four Qwen two point five Coder configurations, twenty-three of them healthy. Thirty-nine HumanEval plus tasks were solved by every healthy one, and the suite is the first fifteen of those. A tag scoring under nine is flagged. It is then compared with its group baseline, the median score of the same model at high precision. A suspect scores at most two where that baseline is at least eight, and a suspect is called non-functional if it also solves under ten percent of all one hundred and sixty-four tasks.

Then comes attribution. For each suspect, the author downloads the original checkpoint and quantizes it at the same level with no calibration, which is the recipe the registry metadata indicates. This plain rebuild is the referee. The two files are compared tensor by tensor, checking type, shape and bytes while ignoring metadata and order. If the suspect fails while the plain rebuild works, that is a file defect. If the rebuild fails too, it is a collapse, and the collapse counts as rescued when a calibrated build of the same weights at the same level clearly beats the plain one. The defect rule was fixed in a dated plan before any rebuild; the rescue rule was added after the first results.

Seven official tags, two point three percent, are non-functional. Six solve none of the one hundred and sixty-four tasks. All are low-bit tags of models with at most four billion parameters: four adjacent levels of Qwen Coder at three billion, one phi three point five mini tag, and two llama three point two tags at one billion. The four Qwen files match the rebuilds in all four hundred and thirty-four tensors and give byte-identical answers. The phi file matches too. None is corrupted.

The comparison that tests the thesis holds weights, toolchain and tensor mix fixed and adds only an importance matrix. The two-bit Qwen level goes from zero to at most one hundred and thirteen tasks. Two of the three-bit levels climb into the seventies and eighties, against one hundred and thirty-two for the registry's four-bit tag. Then the control: the same uncalibrated recipe at one point five billion parameters matches in all three hundred and thirty-eight tensors and stays functional, so the recipe alone does not predict failure. An earlier version of this study chose its referees by distributor, not recipe, and wrongly called these files defective conversions.

There is a threats-to-validity section, and it is specific. The screen tests Python code generation only. The census covers one registry's code-capable tags at or under eight gigabytes, plus a small, popularity-selected community sample. Seven is a lower bound: a deepseek coder group that scores low at every precision, full sixteen-bit included, escapes the count. The rescue evidence comes mostly from Qwen. Generation used one pinned runtime, the rebuilds used a newer llama.cpp than the registry probably did, and the run plan was not publicly preregistered. The two llama tags were never rebuilt, so they stay unattributed. Perplexity itself flagged every broken build measured, at twenty-seven to five thousand seven hundred times the full-precision value, but that threshold was chosen after seeing the results.

The idea to carry away is that provenance and function are different properties. A quantized file can be reproduced exactly from its source weights and still not work, and only running it tells you which you have. The registry served five broken files unchanged, byte for byte, from the August census into October. The proposed gate is cheap: a fifteen-task screen, about thirty seconds of generation per tag, and none of sixty-six sampled tags it passed was non-functional. Screen low-bit tags before publication, then calibrate or withhold the failures, and record the recipe beside the result.