Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load

Instruction Capacity. Behavior and Degradation of Language Models under Controlled Load. By Hennadii Kutomanov, working independently. Published on the twenty-sixth of October.

Give a language model more and more instructions at once, and the quality with which it follows them falls. That much is known. What has been missing is a quantity that states where a model's limit sits, expressed in units of load, and carrying an uncertainty. Without such a number, saying that one model handles heavier prompts than another stays a matter of impression.

The quantity proposed is called instruction capacity: the number of rules a model holds while keeping a given share of its replacements right. It is reported together with its uncertainty. The instrument that produces it also measures how performance depends on each load factor along the way, and records the collapse of the answer as a behavior of its own.

The load is built from rules that replace single-token words in a text that carries no meaning. That choice buys three properties. Each instruction is atomic. The rules are independent of one another by the construction of the task. And the outcome of every application of a rule is fixed by a reference answer that can be computed. Every occurrence of a word due for replacement then becomes a Bernoulli trial, a single success or failure, with a known position in the answer and a known rule number.

A statistical model sits on top of those trials. First it measures the shape of how the probability of a right replacement depends on five factors: the text length, the word position, the replacement density, the number of rules, and the rule number. Then it describes those dependences with smooth laws, and restores a reference level that does not depend on the choice of the coding. From that fitted model comes the instruction capacity, with its uncertainty. The results are gathered into what the author calls a passport for the model, and every number in that passport can be recomputed from the published points. The measurement data and the passport are open.

The instrument was applied to GPT-4o mini. The measurement used two thousand nine hundred and fifty-eight requests.

Those requests produced six hundred and forty-two thousand four hundred and thirty trials. Regular laws emerged for every load factor, and for the collapse of the answer as well. At a share of right replacements of zero point nine five, the instruction capacity came out at about thirty-seven to thirty-eight rules. At the full replacement density, it was about eighteen to nineteen rules. Pushing the density up, then, roughly halves the number of rules the model can hold at that standard.

The ambition here is metrological. A model's passport is meant to serve as the calibration for a future meter of the instruction load in prompts, a tool that could one day say how heavy a given prompt is relative to what a given model can carry. This first measurement, on one small model, shows that the degradation follows regular laws that can be fitted and stated with an uncertainty, rather than an impression that prompts with more rules simply work worse.