LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

LLMs Don't Know What They Know. Benchmarking Temporal Self-Knowledge for Financial Backtesting. By Lingfei Qian and fourteen others, from The Fin AI, the University of Manchester, and four other institutions. Published on the fourth of November.

A financial backtest of a large language model means something only if the model did not already know the outcomes it is being asked to predict. If it saw the answer in training, a good forecast is just a memory. Practitioners guard against this in three ways. They trust the vendor's stated knowledge cutoff. They ask the model what it knows. Or they instruct it to answer as of some past date. Each of those safeguards is taken on trust, and the authors set out to verify whether any of them deserves it.

The tool is FinDendro, a benchmark of two thousand two hundred and twenty-six dated U.S. facts spanning more than a quarter of a century, up to the present year. The facts cover monthly macroeconomic series, month-end stock prices, and Federal Reserve decisions. What it measures is temporal self-knowledge: whether a model's account of what it knows lines up with what it demonstrably knows.

Each fact is turned into four prompts. The first is a plain value question. Then come Recall and Predict questions, which pair each candidate value with a claim either of knowing it or of guessing it, with no date declared. The fourth is a Predict question with a historical date declared. Comparing a model's answers across those four separates what it gets right from what it says about getting it right. Controls for persistence and for dominant outcomes keep chance accuracy near zero point two seven, so a correct answer is not easy to reach by luck or by repeating the last value.

For most frontier models, self-reports understate knowledge. GPT-5 answers seventy-six percent of the controlled facts correctly, but claims to know only thirty-eight percent. And of the facts it says it does not know, it answers sixty-one percent correctly.

The other two safeguards fare no better. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, so where the model believes its knowledge ends is not where it ends. And declaring a past date removes most claims of prior knowledge while leaving accuracy unchanged. The model stops saying it knows, but keeps answering as well as it did.

The conclusion is direct. Inside a model's training window, a backtest cannot be certified leak-free by self-report, by instruction, or by a stated cutoff. What the authors argue for instead is behavioral measurement of temporal knowledge boundaries, and FinDendro supplies both the benchmark and a procedure for choosing backtest periods based on factual knowledge the model has actually demonstrated, not knowledge it admits to.