Generative Late-Interaction Embeddings For Visual Document Retrieval

Generative Late-Interaction Embeddings For Visual Document Retrieval. By Mohamed Eltahir and seven others, from King Abdullah University of Science and Technology and Edge Hill University. Published on the tenth of September. Its code repository is public and real: about twenty source files.

Late-interaction retrieval is the state of the art for searching visual documents, but it pays for its accuracy in storage: each page is represented by about a thousand vectors. Existing compression methods keep a subset of those vectors, or a local average of them. Under aggressive storage budgets, those methods degrade sharply, and the alternatives require retraining the encoder. That is a heavy price when a collection runs to many pages, and it is the cost this work sets out to cut without giving up the accuracy that made late interaction attractive in the first place.

Investigating that degradation across three encoders, the authors found two consistent properties. The vectors lie exactly on the unit sphere. And they concentrate near a manifold whose intrinsic dimension is only five to six. A page's thousand vectors, in other words, have very few real degrees of freedom.

That geometry yields two insights. First, the centroids that standard k-means produces fall inside the sphere, and this makes MaxSim scores systematically too low. Normalizing the centroids back onto the surface is a free correction, worth up to zero point zero nine three in nDCG at five over the raw centroids. Second, because the page manifold has so few degrees of freedom, the full set of vectors can be regenerated from only a few of them. The first insight fixes existing compression for free. The second is the one GLIE is built on: if a thousand vectors are really a handful of coordinates in disguise, it should be possible to store the handful and recover the rest.

Generative Late-Interaction Embeddings, GLIE, builds on both. Each page gets a small number of vectors, far fewer than the original set, learned from the normalized centroids. These serve two roles: a lightweight index, and a basis from which the page's full set of embeddings can be regenerated. At query time, search runs only on these few vectors. A decoder then expands just the top candidates back to their full set of vectors, for exact rescoring. The expensive step, rebuilding every vector, happens only for a short list of candidates, so the storage saving applies to the whole collection while exactness is kept where it matters for the final ranking.

On ViDoRe v1, at four vectors per page, GLIE keeps nearly eighty percent of the uncompressed system's nDCG at five, against seventy percent for the best earlier post-hoc method. The network doing this has four hundred and fifteen thousand parameters. It was fitted in under three GPU-minutes, on just a thousand training pages.

At a matched training budget, fine-tuning the encoder does not reach even the training-free stage of GLIE, and the full system beats it at every budget. The same patterns hold on a second encoder, and on ViDoRe v2. That last comparison matters for practice. The expensive alternative, retraining the encoder, does worse than a small post-hoc network that leaves the encoder alone.

Sampling evidence gives way to reconstructing it. Compression usually means deciding which vectors to throw away. GLIE keeps a compact code and rebuilds the full page on demand, only for the pages that made the shortlist, and the authors see this as a new axis for storage-efficient retrieval, with the decoder as the main place left to design.