EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

EngramEdit. Decoupled Knowledge Updates in LLMs through Conditional Memory. By Hongru Cai and six others, from the Hong Kong Polytechnic University, Hangzhou Diagens Biotechnology, and one other institution. Published on the seventh of October.

Conditional memory architectures, DeepSeek Engram being the example named, use the n-grams in a model's input to look up learned embeddings. That expands the capacity of a large language model at the price of only limited extra computation. Beyond scaling, the architecture has shown it can separate where factual knowledge is stored from the general-purpose computation the Transformer performs. That suggests a way to update facts while leaving the Transformer backbone fixed.

Realizing it is hard for two reasons. A single fact can be expressed in different ways, and different expressions may activate different n-gram embeddings, so changing one lookup may not change them all. And the embeddings are shared: updating one can unintentionally change what the model predicts about other facts.

EngramEdit is a method for decoupled knowledge updates that works through the conditional memory rather than through the backbone. It answers each of the two difficulties in turn. Against the problem of many expressions, it targets the fact across multiple phrasings at once. Against the problem of sharing, it is deliberately conservative with the embeddings that many inputs depend on.

The procedure has two steps. First, EngramEdit computes target memory representations: the memory states that would make the model predict the updated fact across multiple expressions of it. Second, it jointly updates the shared n-gram embeddings to match those targets, across expressions and across edits together. During that update, changes to frequently reused embeddings are penalized more strongly, which is how unrelated knowledge is protected.

Experiments show that EngramEdit enables independent factual knowledge updates through conditional memory, with near-perfect editing success. The revised knowledge is usable across expressions the method never saw, and in multi-hop reasoning. Under chain-of-thought prompting, its accuracy is nearly three times that of the strongest baseline. Unrelated knowledge and general capabilities are largely preserved, even as factual updates accumulate.

The authors present this as a change in what conditional memory is for. It began as a way to scale a model cheaply, adding capacity through lookups rather than through more computation. Its separation of stored facts from general computation was a promise, and the two obstacles were what stood between that promise and practice: facts spread across many phrasings, and embeddings shared between many facts. The joint update across expressions answers the first, and the stronger penalty on heavily reused embeddings answers the second. With EngramEdit, the same memory becomes an editable knowledge interface: a place where facts can be revised independently, while the Transformer that reasons over them is left untouched.