Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

Keyword Harnesses Fail Open. A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models. By Juan S. Santillana, working independently. Published on the first of October. No code repository is linked to it.

Keyword-matching benchmarks can give a small model credit for tool use it never actually performs. The paper documents one such false positive and proposes a ladder of diagnostics that are strict and cheap enough to catch it. The case is a matched-architecture pair of Spanish security language models. They share a decoder, a tokenizer and special tokens, and on lenient tool-use metrics they score almost identically: zero point six six zero against zero point six five zero on the measure called B4.

Their histories differ. The smaller model has six hundred sixty-one point six million parameters, was trained on roughly sixty-five percent code and technical text, and had no dedicated supervised fine-tuning. The larger, at about one point one billion, went through a web-heavy multi-phase curriculum followed by six billion tokens of tool-focused fine-tuning.

The first rung of the ladder is a verbatim-reproduction check on training examples, and it separates the two completely. The six hundred million model emits valid tool calls with generalised arguments on six of six examples. The one billion model does so on none of six, across checkpoints. A first-token probe then localises the failure to a missing prior: the larger model puts probability between ten to the minus four and ten to the minus five on the tool call token, a prior erased by its web-heavy training phase.

A targeted fine-tuning recipe repairs it, with a diverse corpus, a five times higher learning rate, two thousand two hundred and two steps and about three point three GPU-hours. That is three orders of magnitude fewer tokens than the phase that failed. Across all two hundred and sixty-nine corpus rows, valid emission rises from zero point one to zero point nine five nine. On two hundred and thirty-eight unseen prompts, the repaired model passes zero point five three six of the time against the smaller model's zero point four two eight, with a p-value of zero point zero zero four.

Embedding-drift checks show the repair left the trigger token's tied embedding where it was. Ninety-seven point seven percent of the bf16 table stays bit-identical, so the changes live in the surrounding network.

Several limits are stated plainly. Both models over-trigger, rarely answering a negative prompt without making a call: zero point zero nine of the time for the smaller model and zero point one seven for the repaired one. Factorial analyses confirm that every repair configuration installs the format, but the idea that a diverse corpus helps suppress unwanted calls stays a hypothesis, because the results are sensitive to the random seed. And the evidence comes from a single matched pair of small models.

Two models that a lenient harness scores as equals turn out to sit at opposite ends: one calls tools, the other almost never could. The ladder that exposes this costs minutes of CPU time, and the author's position is that checks of this kind should gate any tool-use claim made about a small model.