Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters

Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters. By Tardelli Ronan Coelho Stekel, from the Federal Institute of São Paulo. Published on the twenty-second of October. Its code repository is worked out: about sixty-five source files, with a readme and a licence.

Many language-model pipelines end in a closed-set decision, such as picking which tool to call. Prompting a large language model for that choice is slow, and its answer is sensitive to the order the options arrive in. The paper asks two questions. How small can an encoder for these decisions be? And should it read the options jointly, all at once, or separately, scoring one option at a time? The central claim is that per-option scoring wins, but only under one condition: the encoder has to start from a reranker initialisation.

The core study runs three seeds at thirty-two million parameters. On a held-out sample of a decision suite built from thirty-seven benchmarks, per-option scoring beats joint scoring by five point eight points of chance-corrected skill when optimizer steps are matched. When training compute is approximately matched instead, the margin is two point eight points. Two external benchmarks agree. Crossing the reader with the initialisation shows where the advantage comes from. A reranker takes a query and a candidate as a pair, which is exactly the shape of per-option reading. Start from a reranker, and per-option wins. Start from the raw encoder, and the two readers cannot be told apart.

The paper also releases six joint encoders, from seventeen million to one billion parameters. Two of them, the sixty-eight million parameter encoder and a larger sibling, beat a DeBERTa-v3 zero-shot classifier that is bigger than either. The margins are four point nine and five point nine accuracy points, on two hundred and nine System1-Arena tasks audited for overlap. On the one hundred and forty-five tasks unseen in training, they are at least on par with it. The smallest encoder answers a median request in fourteen milliseconds on sixteen processor threads. The classifier takes seven hundred and twenty-eight.

Each reading has its price. Per-option scoring needs five to thirteen times more encoder passes. In latency that comes to at most twice the median, but more than four times the mean, so the slow tail grows far faster than the typical request. The joint models carry the opposite weakness: reordering the options changes twelve to twenty-three percent of their answers, while per-option scoring changes none. The external benchmarks agree only in direction on their unseen tasks, where the difference is not significant. And the leaderboard result has its own boundary. Up to sixty-eight million parameters, the joint models lead their size range on a public leaderboard's private tests, which are mainly in-distribution.

Put together, the study turns a design choice into a trade with known terms. Reading options one by one buys accuracy and complete immunity to option order, provided the encoder began life as a reranker, and it pays in encoder passes and tail latency. Reading them jointly is cheaper, but the order of the list leaks into the answer. Either way, the decision fits in a compact encoder that answers in milliseconds on ordinary processor threads.

Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters. By Tardelli Ronan Coelho Stekel, from the Federal Institute of São Paulo. Published on the twenty-second of October. This is the long version: the whole paper rather than the abstract.

Many language model pipelines end in a decision from a closed set: which tool to call, whether a retrieved passage supports a claim, which intent a message expresses. Prompting a large model for it is slow and sensitive to the order the options are listed in. A natural language inference model needs one forward pass per option, and a fine-tuned classifier does not transfer to a new label set. A recent format packages such problems as typed decisions: a state and some typed questions go in, a probability for every allowed option comes out, and no text is generated. Because the output space is fixed, the model can in principle be small. The paper, by a single author at the Federal Institute of São Paulo, asks how small, and how the encoder should read the options.

There are two classical ways to read the options. Per-option scoring encodes the context once for each option and normalises the scores across them. Joint scoring puts the context and every option into one sequence, with a marker token before each option, and reads one score per option from those markers. Joint scoring costs one pass per question instead of one per option, which is why recent compact decision models use it. The two had not been compared under controlled conditions for this kind of decision. One more fact matters later. The models start from Ettin rerankers, cross-encoders trained on pairs of one query and one document.

The released family is called Crivo: six joint encoders from seventeen million to one billion parameters, each fine-tuned from the Ettin reranker of the same size. Each question becomes one sequence: an instruction, the state, then each option preceded by a new marker token. The hidden state at each marker, plus a learned embedding of the question type, passes through a small head to a single logit, and a softmax across the options gives the probabilities. Options attend to each other. The loss is cross-entropy plus half the Brier score, and the smaller models also learn from a larger teacher. All six took about eighty dollars of rented compute.

The heart of the paper is a reader study at thirty-two million parameters, on a fixed subset of one hundred and fifty thousand questions, with three seeds per arm. It has a trap. Because each per-option sequence holds a single option, the per-option arm takes about three point two five times as many optimizer steps on the same rows. So joint controls are added, the primary one training the joint reader for exactly as many steps at its usual batch, which roughly matches compute by repeating rows. A two by two grid then crosses reader with initialisation: reranker or raw encoder. The plans and decision rules for these controls were written in a dated log before the runs.

At matched steps, per-option reading leads by five point eight points of chance-corrected skill on a held-out sample of the public suite. Matching steps alone barely helps the joint reader. Matching compute closes about half the gap, and per-option still leads by two point eight points. Both external benchmarks point the same way, but on their unseen tasks the difference is not significant. The two by two grid is the finding the abstract only hints at. From the raw encoder the two readers are indistinguishable. The reranker helps the per-option reader on all three evaluation sets, and the joint reader on only one. The paper's most direct reading is a format match: a reranker was trained on exactly the per-option input. It adds that the grid does not test that mechanism. The gain is also concentrated: the median benchmark gains under one point, and five benchmarks carry most of the mean. The price is five to thirteen times more encoder passes: about the same median on the graphics card, but a mean four point three times higher, and twenty-seven times higher on the processor.

On all two hundred and nine System1-Arena tasks, the sixty-eight and one hundred and fifty million models beat a four hundred and thirty-five million parameter DeBERTa zero-shot classifier by about five and six points. On the unseen tasks the margins shrink to about three points and do not survive a multiple-comparison correction, so the paper reads them as parity. The smallest model answers a median request in about fourteen milliseconds on a processor. DeBERTa takes seven hundred and twenty-eight.

The paper has a limitations section, and it is specific. The reader study uses one subset and one size, and the compute control repeats rows, so a joint arm on more distinct rows would be stricter. With three seeds per arm, intervals that barely exclude zero are treated as suggestive. The models are over-confident on new tasks: on unseen System1-Arena tasks every Crivo has a Brier score no better than a uniform guess. Training and evaluation are English only, and the closest encoders built for this format were not run on the external benchmarks.

Two negative results go further. The development panels overestimated every model's private leaderboard score by five point six to seven point one points and ranked two recipes in the wrong order, because the new recipe drew on the same pool as the panels. And Prumo, a rule engine with no learned parameters, ties the smallest model on the leaderboard's same-skill private tests while staying near chance on the external benchmarks: part of that board rewards format-specific solving.

The question in the title has a conditional answer. Reading options separately is not better in itself. It is better when the starting checkpoint was trained on the same pairwise input, and otherwise the cheaper joint reader is as good. The practical choice reduces to two questions: is a pair-trained checkpoint available, and how heavy is the tail of your requests? A second lesson travels further: a development set drawn from the same pool as your newest training data has stopped being an independent check.