ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport

ColNanoVDR. Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport. By Zhuchenyang Liu and three others, from Aalto University and Independent Researcher. Published on the twenty-eighth of September. It links to a code repository that is no longer there.

Multi-vector retrievers built on vision-language models lead visual document retrieval, but they run a query encoder with billions of parameters on every search. Distilling that encoder into a small student, one that queries the teacher's existing index, would remove the bottleneck. The standard recipe, though, matches the teacher's MaxSim scores, which means encoding and caching every training page, and that can reach terabytes of page tokens. An earlier approach, NanoVDR, avoided pages entirely by training on the teacher's query embeddings alone, but only for single-vector retrievers. Multi-vector models represent a query as many token embeddings rather than one, and that is exactly where the page-free trick had not yet been made to work.

ColNanoVDR brings that document-free distillation to multi-vector retrieval, a first as far as the authors know. Its objective is called OTW, for Optimal Transport with Learned Weights. It aligns the student's query tokens with the teacher's through entropic optimal transport, giving each student token a learned weight, and it needs no correspondence between the two tokenizations. That last property matters because a small student and a large teacher need not split a query into the same pieces. With this objective, nobody has to decide in advance which student token answers to which teacher token, and each student token carries a weight of its own that is learned rather than fixed.

The guarantee behind it is formal. The authors prove that the resulting alignment cost bounds the difference in MaxSim score on every page. Keeping that alignment cost low therefore keeps page scores close, and the pages themselves never have to be touched during training.

Students are distilled from five state-of-the-art teachers. At one hundred and forty-nine million parameters and text-only, they retain about ninety-five percent of their teachers' NDCG at five on ViDoRe versions one through three, while encoding queries up to twenty-six times faster.

The more direct comparison isolates the objective. Under identical training, OTW matches score distillation while encoding no page at all and reading twelve point six times less cached teacher data.

The shift is in what gets compared. Score matching compares pages, and so needs the pages. ColNanoVDR compares the queries themselves, and a bound carries that comparison back to every page in the index, which is where the terabytes of cached tokens drop out.