Periscope: Extending Frozen Language Models Beyond Their Context Window

Periscope. Extending Frozen Language Models Beyond Their Context Window. By Mohamed Eltahir and seven others, from KAUST and Edge Hill University. Published on the second of October. Its code repository is public but slight: nine source files.

A language model reads a long text in one forward pass whose cost is quadratic. It stops at the edge of its context window, and it loses accuracy with length well before it reaches that edge. The authors ask a narrower question than how to read everything at once. When the decision is over a finite set, which document is relevant, which option is supported, which passage is the evidence, can the read be factorized?

Periscope says it can, at inference time and without training. The text is cut into chunks, and the chunks are laid out on a square grid, K by K. A frozen model is then asked the same question about two kinds of span. There are K local spans, each a run of consecutive chunks, which look closely at one stretch. And there are K strided spans, each sampling across the whole text, which look broadly at all of it. For every span, the model's log-odds for every answer are read at a single token.

Each answer keeps its best local score and its best strided score. Then comes the by-product the method leans on. Every chunk belongs to two spans, one of each kind, and scoring each chunk by its two spans yields an evidence map at no further cost. The peak of that map is the chunk behind the answer.

The arithmetic is what lets it scale. Each probe is short compared with the whole text, and depends on the text length and the chunk size. A model whose window is W tokens can therefore reach a text of about W squared over the chunk size, while the cost grows with text length to the power one point five rather than quadratically. In the authors' words, the map replaces the long read.

On LongBench v2, reading only the K chunks the map ranks highest, about nine thousand tokens, matches the same model's best window read, across windows from thirty-two thousand to one million tokens. On InfiniteBench, where the median context is one hundred and fifty thousand tokens, Periscope leads the best window read by five points. And on the long-document corpora of BRIGHT, the same map gives the best NDCG at ten of six methods.

Memory is the other half of the story. Each call caches only one probe. That lets a twenty-seven billion parameter model read contexts of four point five million tokens on one eighty gigabyte GPU. A single pass over that text would need two hundred and ninety-six gigabytes of cache.

Periscope's last sentence states the shift plainly, and there is little to add to it. A long read then needs a GPU that holds the model, not one that holds the text. The length of what can be read stops being a property of the hardware and becomes a property of how the question is asked.