CoWindow Attention: Full Causal Coverage Is a Collective Property

CoWindow Attention. Full Causal Coverage Is a Collective Property. By Jingze Shi and eight others, from the Hong Kong University of Science and Technology in Guangzhou, the Beijing Academy of Artificial Intelligence, and one other institution. Published on the twenty-sixth of September. Its code repository is worked out: about nine hundred source files, with a readme and a licence.

Full attention, which the abstract calls FullAttn, shows the complete causal history to every attention head, over and over again. Each head sees every earlier token. That repetition creates a large amount of redundant computation and memory traffic, and the abstract is careful to add that this holds even when the dense kernels doing the work are already IO-efficient. The waste is not in how the computation is carried out. It is in how much of it is duplicated across heads.

CoWA, short for CoWindow Attention, is a structured attention architecture that spreads access to the causal history across the KV heads instead of handing all of it to each one. The claim in the title is the heart of it. Full causal coverage does not have to be a property of every head individually. It can be a property of the heads taken together.

The layout is fixed by position. All heads share two windows: a near-diagonal window over the most recent tokens, and a prefix-sink window at the start of the sequence. The rest of the history is covered by long-range windows that are complementary, meaning they partition what remains among the heads. Put the heads side by side and their union covers the full causal past, even though any single head attends only sparsely to distant tokens. Because the pattern is defined by position, there is no learned router and no indexer. The same pattern is used in training and in inference, and it lines up with tensor parallelism over KV heads.

A window-matched ablation at eight thousand tokens isolates the complementary long-range allocation. With one hundred percent collective coverage, CoWA reaches eighty-nine point seven three percent accuracy, against eighty-nine point nine seven for FullAttn. Giving the heads duplicated long-range windows instead does substantially worse. In a broader associative-recall comparison with matched token budgets, CoWA closely tracks FullAttn as context grows, while other sparse patterns lose a substantial fraction of the associations.

Speed is measured on an attention-operator benchmark at one hundred and twenty-eight thousand tokens with tensor parallelism. In training, CoWA cuts forward latency by seven point four times and backward latency by eight point six times. Decoding latency in inference falls by three times. Peak operator memory per rank matches FullAttn in training and is seven point six times lower in decoding. Scaling-law training runs from zero point six billion to fourteen billion parameters, and CoWA tracks FullAttn in perplexity while spending fewer total training FLOPs. The resulting fourteen billion parameter models, along with thirty-two billion parameter models from separate continued training, score comparably to FullAttn on knowledge, reasoning, and long-context retrieval.

One picture holds the whole design. Dense attention gives every head a full view of the past. CoWA gives each head a shared view of the near and the very beginning, plus its own slice of everything between, and lets coverage emerge from the ensemble. The reported accuracy stays close to FullAttn while latency and decoding memory drop, which is the evidence offered that full coverage can be collective rather than duplicated.