Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Predictive Credit. Measuring What Scientific Explanations Add to Experimental Forecasts. By Jingjie Ning and three others, from Carnegie Mellon University and Tsinghua University. Published on the twenty-ninth of September. No code repository is linked to it.

Research agents explain the experiments they plan. The question here is whether those explanations carry any predictive value. If you know why an experiment is expected to work, can you forecast its outcome any better? The paper calls that quantity predictive credit, and sets out to measure it rather than assume it.

The measurement rests on paired forecasts. Each pair shares the same intervention, the same forecaster and the same outcome, and varies only what the forecaster is shown: a plain description, a matched explanation, or donor context. Five checks are tracked along the way. Commitment, delivery, predictive gain, alignment, and uptake of a signal that is already known.

The protocol runs across three hundred and thirty-six prospective states, spread over controlled learning, twelve Tox21 endpoints, and twenty-four OpenML tasks. The decision about credit was frozen in advance, in what the abstract labels v5, and each setting carried its own preregistered rule. For Tox21 that was a harm test on interval score for ROC AUC. For OpenML it was a joint rule covering formation, point-equivalence and repeatability.

The frozen decision came back inconclusive. The Tox21 harm test was not met: the difference was minus zero point zero zero two six, with a ninety-five percent interval running from minus zero point zero one seven four to zero point zero one zero four. The OpenML rule was not met either. Matched gains in point accuracy over description stayed unconfirmed, and the seed-donor intervals for both Tox21 and OpenML spanned zero.

Secondary results pull in different directions. Under the requested DeepSeek V4 Pro, matched cards reduced secondary Tox21 drift by sixty-four point five percent, and donor cards by fifty-nine point one percent. A replay on DeepSeek V4 Flash went the other way. It raised matched point MAE from zero point zero one eight two three to zero point zero two zero two, and missed equivalence on interval score between matched and donor.

On OpenML, assigning the full card widened nominal eighty percent intervals by twenty-one percent. Coverage was forty-nine point three percent, against fifty-one point four percent for description, with content present in sixty-six of the one hundred and forty-four cards. Direct-text Flash delivered all one hundred and forty-four notes with no detectable gain in matched point accuracy. One positive control did register. A mechanism written by a researcher lowered point MAE by two point six percentage points against description.

What the paper delivers is a protocol rather than a verdict. It measures predictive credit for research-agent benchmarks and for scientific forecasting, and at the donor resolutions tested, credit for natural explanations remained unconfirmed. A mechanism a researcher wrote by hand did move the forecasts. The explanations under test have not yet been shown to.