Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing

Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing. By Ding Wu and three others, from the Georgia Institute of Technology, the University of Pennsylvania, and two other institutions. Published on the second of October. No code repository is linked to it.

Large language models are used more and more as general interfaces to factual knowledge, but their parameters do not update themselves when the world changes after pretraining. Knowledge editing offers a targeted alternative to costly retraining: modify the selected knowledge, and preserve everything unrelated along with general capability. Conventional editing works on structured factual triples. Unstructured knowledge editing, or UKE, works on free-form passages that hold several facts at once.

Existing UKE editors share a failure mode called context reliance. An edited model can often reproduce the editing passage, yet fail to recall the individual facts in it reliably once the original passage is not there as context. Alongside it, the authors identify a problem in the standard editing objective, which works at the level of the whole passage. Facts later in a passage receive increasingly rich ground-truth context ahead of them, so they start with lower losses, and that makes them look easier to learn than they are. The authors call this context-induced difficulty underestimation.

FOVEATED is a plug-and-play framework that gives each sentence a focused view. It builds those views by randomly shifting the Rotary Position Embedding positions, RoPE for short, that are assigned to the keys of the context preceding the sentence. The name echoes the eye's point of focus: one sentence at the centre, and the context before it moved about.

The perturbation exists only during editing. Afterwards it is removed, so at inference time the model's native positional encoding is exactly as it was. FOVEATED is instantiated for two families of editor: those that optimise directly, and those that first locate and then edit. The authors also give a theoretical analysis of how the method counteracts the underestimation of difficulty that the passage-level objective produces.

Empirically, the improvements are consistent across five knowledge editors, two language model backbones, and three benchmarks. The target throughout is recall of individual atomic facts, the very thing context reliance had been undermining.

The diagnosis carries the weight. A passage-level objective quietly hands later facts the earlier ones as support, so their low initial losses make them seem easier than they are. Focused views shift where that support appears to sit while editing, and only while editing, which is how the fix reaches the training signal while leaving the edited model's ordinary behaviour at inference untouched.