FinForesight: A Diagnostic Backtesting Benchmark for Deep Research Systems in Financial Forecasting

FinForesight. A Diagnostic Backtesting Benchmark for Deep Research Systems in Financial Forecasting. By Fengyuan Liu and seven others, from the University of Hong Kong, the Hong Kong Polytechnic University, and four other institutions. Published on the first of November.

Forecasting financial outcomes means synthesizing evidence as it evolves and reasoning under uncertainty. That combination makes deep research systems look like promising forecasting tools. Measuring how good they actually are is the hard part. A historical backtest asks a system to predict something that has already happened, so it can end up rewarding outcomes the system memorized, or information it retrieved from after the forecast date. Live forecasting closes that loophole, since the outcome does not yet exist to be looked up. But the feedback only arrives once events resolve. One approach is quick but leaky, the other clean but slow.

FinForesight is a diagnostic backtesting benchmark for financial deep research systems, meant to deliver timely feedback while still letting historical scores be assessed for how far they can be trusted. It spans the Chinese, U.S., and Hong Kong markets and covers four forecasting tasks: valuation, return ranking, event direction, and future financial fundamentals. Two further diagnostic tasks probe counterfactual reasoning and financial calculation. Those two are scored separately from forecasting performance, so the capabilities underneath a forecast can be read apart from the forecast itself.

The framework is compatible with existing harnesses, and it restricts retrieved evidence to information that was available on the forecast date. To test whether controlled backtests of this kind approximate future performance, the authors compare three regimes side by side: evaluation on the pretraining era, evaluation on questions resolved after the model's cutoff, and live evaluation.

In most temporal comparisons, pretraining-era scores come out higher than post-cutoff scores. Stronger retrieval controls narrow the gap between post-cutoff backtests and live results, bringing the cheaper measurement closer to the slower one. Forecasting attribution and robustness analyses then show how scores respond to the retrieval policy, and how predictions shift under prompt rewrites and under numerical perturbations.

The benchmark's premise is that a backtest is only as trustworthy as its boundary on time. Here that boundary is a date limit on evidence, checked against live results, and tightening it moves the backtest toward what live forecasting reports.