Going Deeper With Memories

Going Deeper With Memories. By Afshin Khadangi, from the University of Luxembourg. Published on the third of November.

Byte level language models have two real attractions. They avoid any dependence on a tokenizer, and they provide universal coverage, since every input is ultimately bytes. The price is cost. Processing every single byte at full resolution is expensive, and that expense is what has to be addressed if the attractions are to be kept.

DeepMem is a causal multiscale architecture built on one separation. Semantic compression is one concern. Exact preservation of bytes is another. Rather than asking a single pathway to do both, the architecture keeps the two apart, so that compression can be aggressive without the exact bytes being lost along the way.

A learned invertible subband transform builds a hierarchy of shorter and shorter sequences. Alongside it, a lossless carrier and explicit detail sidebands preserve byte identity. Fast weight memory blocks operate along the encoder, and at the compressed bottleneck the state is computed iteratively, with that computation shared. Finally, an attention based super resolution decoder restores representations at byte resolution, bringing the model back from the short sequence to the full one.

Against a matched byte level Transformer, DeepMem achieves stronger language modeling on FineWeb-Edu and on PG19, and the best macro average across the evaluated FLORES languages. It also improves PIQA accuracy, while staying competitive on code modeling, multiple choice reasoning, and robustness to byte level corruption. Then comes the efficiency figure. Refinement at the bottleneck needs thousands of times fewer counted FLOPs than equivalent reasoning at full resolution, while evaluation throughput stays competitive.

The question DeepMem answers is where the expensive thinking should happen. Its answer is the short, compressed sequence, with the exact bytes carried alongside so that nothing is lost on the way back to full resolution.