Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?

Beyond Benchmarks. Can Small Open-Weight Models Get Real Coding Work Done?. By Vaibhav Jain, working independently. Published on the twenty-fifth of October. No code repository is linked to it.

The central claim is stated in the first line: making an open-weight language model generate text is not the same as making it finish coding work. The author frames this as a retrospective, empirical systems experience report. It follows the author's own engineering work from July to October, across coding clients, model-specific serving recipes, and the runtimes vLLM, SGLang, llama.cpp and MLX.

Two observations organise it. Models behind an endpoint that was nominally compatible still differed in tool-call formatting, in turn control, in context behavior, in cache reuse and in reliability. And progress came in distinct levels that should not be confused: a model being served, the model producing coherent text, tool calls actually executing, and code that passes independent behavioral checks. Reaching one level did not mean reaching the next.

The evidence base has three parts. There are four thousand one hundred and sixty-five coding-agent gateway turns from the author's own sessions. There are six hundred and twenty-three debug jobs. And there are three separate delegation studies, each with hidden behavioral checks and a fixed Claude Opus 5.5 parent.

Caching shows up sharply. Saved cold-prefix requests had a median time to first token of about sixty-nine point four seconds. High-reuse requests had a median of two point eight seconds. The author marks that comparison as observational.

In a forty-five-trial study, direct coding passed fifteen out of fifteen. Delegating to a quantized Qwen3-Coder worker passed twelve out of fifteen, and in the other reported result, fourteen out of fifteen.

A second suite of fifteen tasks ran under a fixed artifact contract. In the completed replacement cohort, direct coding scored forty-five out of forty-five. A Nemotron Cascade 2 worker scored twelve out of forty-five, with eleven worker substitutions. Sixteen sessions had stopped on a subscription limit before any model call and were replaced. Counted as failures instead, the scores become thirty-seven and eight, each out of forty-five. A Qwen3-Coder worker passed none of the forty-five.

In both studies that had a direct arm, direct coding came out cheaper per verified success, measured as frontier cost at API list-price equivalents. A post-hoc diagnostic also found that nineteen of thirty-six artifacts rejected only because parameters had been renamed would have passed the hidden checks unchanged. They are still not counted as successes.

The author is careful about what this is and is not. The studies are not pooled together, and they do not rank models or runtimes. The contribution is an evidence-qualified account of five practical concerns: serving, agent protocol, session continuity, artifact validation, and verification. Much of the data comes from one engineer's own sessions over a few months, and the timing comparison is explicitly observational.

What the report hands a practitioner is a ladder of distinctions. A model that is up, a model that talks, a model whose tool calls run, and a model whose code passes are four different milestones. In these studies, small open-weight workers often stopped somewhere on that ladder, while direct coding by the frontier parent reached the top more reliably and, by the author's accounting, more cheaply.