Inverting Multi-Vector Visual Document Indices

Inverting Multi-Vector Visual Document Indices. By Zhuchenyang Liu and two others, from Aalto University. Published on the seventh of October.

Multi-vector visual document retrievers, the prevailing kind, store each page as roughly a thousand patch vectors, often in a vector database that a third party runs. Nobody can read a page from its vectors, so the index is easily treated as less sensitive than the page itself. The authors argue that this is a mistake. The index keeps one vector per patch, in raster order, and every vector is computed by a vision-language model that was pre-trained to read documents. Their hypothesis is that anyone who runs the store, or breaches it, can reproduce a page from the index alone. The page is unreadable to a person looking at the numbers, but the numbers were made by a reader, and laid out in the order the page was read. The conclusion they reach is that the stored index should be protected like the documents it encodes, not treated as a harmless by-product of them.

The attack is framed as conditional document image generation. From the vectors themselves, it infers what it needs: which encoder was used, the shape of the page, and, if the vectors have been shuffled, their original order. On the ViDoRe v3 benchmark, pages inverted from raw indices recover forty-seven percent of the words and forty-five percent of the sensitive tokens. Used as queries against the stored indices, those reconstructed pages rank their own source page first ninety-eight point four percent of the time. Those are two different kinds of leak. Recovering under half the words is partial, but matching a reconstruction back to its exact source almost every time means the index identifies the page, even where it cannot fully spell it out.

Two cheap protections were tested: token pooling and shuffling. Each cuts word recall to about eight percent. Shuffling, though, does not hold. A model that restores the order of a shuffled index lifts the share of source pages ranked first from three point eight percent to ninety-three point five percent. For a pooled index, inversion remains an open problem.

The generalisation test is deliberately simple. The same attack, unchanged, is applied to a different multi-vector retriever. Its inverted pages still rank their source page first seventy point two percent of the time, but here word recall stays below a nearest-neighbour baseline. So the evidence that pages can be identified carries across retrievers more strongly than the evidence that their words can be read back. The results are on ViDoRe v3 and two retrievers, and pooling is the one protection the attack does not yet defeat.

The point turns on an assumption about what an index is. Because the vectors are unreadable to a person, they feel like a summary rather than a copy. The measurements here say that a store of patch vectors, kept in reading order and produced by a model trained to read, holds enough of the page to rebuild a good part of it, and to find the original almost every time.