Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Competence-Gated Pooling of Language Models and Priors for Event Forecasting. By Aditi Tiwari and two others, from the University of Illinois Urbana-Champaign. Published on the tenth of September. No code repository is linked to it.

In hybrid forecasting, a language model is often just one signal among several. A system may already hold a forecast from a market, a crowd or a statistical model, and has to decide whether the language model adds useful information or should be ignored. The relevant target is therefore not how accurate the model is on its own. It is relative competence: the model's marginal value beyond the external forecast already on hand. A model can be quite accurate and still add nothing, if it only agrees with the market. It can also be less accurate overall and still help, if its disagreements happen to point the right way.

Under Brier loss, the authors characterize when disagreement from the model can improve an external forecast. They also derive the gain from using pooling weights specific to each domain, rather than one global weight. The intuition is that a model's added value need not be the same everywhere: it may help in one kind of question and only add noise in another. That analysis leads to a competence gate.

The gate estimates source weights at the domain level from outcomes that have already resolved. It shrinks uncertain estimates toward a global weight, so a domain with few resolved questions does not get an extreme weight on thin evidence, and then recalibrates the pooled forecast. How much a domain trusts the model is set by how the model has actually done against the external forecast there, measured on outcomes.

Across two thousand three hundred and fifty-seven resolved binary questions and five language models, the gate improves the main external baseline from zero point zero seven seven one to zero point zero seven three two in Brier score, and significantly outperforms global forecast combinations. The gain stays significant under leakage controls, and against a leakage-safe time-series prior on the pooled structured set, with separate evidence on FRED.

There is a boundary too. On the official ForecastBench market subset, the gate gives no significant improvement, and there it largely defers to the market. Across four Qwen models, verbal confidence does not reliably identify when the model beats the external forecast. Competence estimated from outcomes, on the other hand, supports better decisions about when to abstain.

The upshot is selective use. Rather than asking whether a language model forecasts well, the gate asks whether it forecasts better than what is already available, domain by domain, judged on resolved outcomes, and gives it weight only to that extent.