On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

On the Tip of the Tongue. First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs. By Jaewon Sok and five others, from Seoul National University. Published on the sixteenth of October. No code repository is linked to it.

When a large language model answers a factual question wrongly in a closed-book setting, the usual explanation is that it lacks the knowledge. Yet models often encode answers they fail to produce. The claim here is that many closed-book errors are something narrower: first-token recall failures. The model commits to the wrong first token. Hand it the right one, and it completes the answer, and often recognizes that answer as correct. That is the title's image: the knowledge is on the tip of the tongue, and the first step is where it slips.

The tool is First-Token Cueing, a behavioral probe that prefills the model's response with the first token of the gold answer and lets it continue. Two observations point at that first token. Model uncertainty peaks there, and the uncertainty at that position already predicts whether the answer will be correct. And among gold prefixes of any length, the first token alone accounts for the largest share of the errors that get recovered.

Across sixteen open models, that single token corrects between a fifth and a half of the errors on multi-token answers in English question answering, and it improves accuracy in all seven languages of a multilingual benchmark. The authors then check that the corrected answers reflect knowledge rather than a lucky hint. Most of the errors it fixes are also solved by the model's own sampled reasoning, which they use as a criterion for encoded knowledge. And the models recognize correct starts they do not produce: they continue a correct start more confidently than an incorrect start of the same answer type and prior probability, yet rarely arrive at it without the gold answer.

The boundaries are explicit in the framing. This is a behavioral probe, so it shows what the model does when cued, not how the knowledge is stored inside it. The effect is for many errors, not all of them: a fifth to a half on multi-token answers in English. The cue itself uses the gold answer, so it locates where errors are recoverable rather than providing a way to recover them without the answer. Missing knowledge remains a real cause of some mistakes; the claim is that it is not the whole story.

Where should a fix aim, then? According to these results, at the first decoding step. That is where a recoverable class of factual errors lives, and the authors point to it as the place where inference-time methods may gain the most.