From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

From Retrieval to Typed Decisions. Calibrated System One Models from Biomedical Sentence Encoders. By Pritam Deka, from Queen's University Belfast. Published on the first of October. Its code repository is worked out: about forty source files, with a readme and a licence.

A typed decision model answers a schema-constrained question about a text in a single forward pass, and returns probabilities that are meant to be thresholded. The question here is whether biomedical sentence encoders, which were trained for retrieval, make good starting points for such models. The answer that emerges is conditional. Retrieval training helps before fine-tuning, and after fine-tuning whether it helps depends on the kind of decision head placed on top.

To test this, the paper builds three things. SBERT2S1 converts Sentence-Transformers encoders into decision models of three kinds: a bi-encoder, a cross head, and a prior-fused residual head, called PFR. BIODECIDE is a suite of biomedical typed decisions. And MEDLINE-S1 supplies two hundred and forty-three thousand training decisions derived from NLM indexing.

Across six parent-retriever pairs, retrieval training improves zero-shot matching of options that carry content. After fine-tuning, the picture splits. Across five pairs and three training-set sizes, retrieval training significantly helps PFR, the head that keeps the retrieval prior, in ten of fifteen comparisons. For the cross head it helps in one comparison and hurts in five.

A matched grid of two heads and five training objectives adds two more results. The cross head beats PFR under every objective. And the released RLCD recipe for open System One models trails cross-entropy by two and a half to three points. Most of that deficit traces to its reward normalisation, which inflates the noisy score-function term by between three point six and fifteen fold. An unbiased leave-one-out estimator recovers most of the gap.

These results are for biomedical encoders, on a biomedical decision suite, with training data drawn from one indexing source. The benefit of retrieval training is tied to a head that keeps the retrieval prior, not to fine-tuned decision models in general. Calibration also has a limit attached: after temperature scaling, no objective is clearly better calibrated than cross-entropy, so the comparison of objectives does not translate into a calibration advantage for any of them.

Whether a retrieval encoder is a good start turns out to be the wrong question asked on its own. The better one is which head it is feeding, since the same pretraining helps one head and hurts the other more often than it helps. The code, the MEDLINE-S1 labels and a model are released.