RenderRank: Learning to Rerank Text with Compressed Visual Tokens

RenderRank. Learning to Rerank Text with Compressed Visual Tokens. By Seongtae Hong and four others, from Korea University and Sookmyung Women's University. Published on the twenty-eighth of September. No code repository is linked to it.

Rendering a document's text as an image lets a vision-language model encode the document as visual tokens, which can make the input shorter than feeding in the text directly. That saving matters most in reranking. Every query means scoring several candidate documents, so whatever is saved on one document is saved again on each candidate evaluated. Conventional rerankers read the full sequence of text tokens every time. A shorter input per candidate, multiplied across a whole candidate list, is where the efficiency case for rendering text becomes strongest. The open question is whether a relevance judgment made from a compressed picture of the text can be as accurate as one made from the text itself.

RenderRank is a reranker that learns to score relevance conditioned on the query, working from compressed visual representations of documents instead of the text token sequences that conventional text-based rerankers use. The document arrives as a picture of itself.

Training has two steps. First, relevance scores computed from the visual inputs are aligned with the scores of a text-based teacher, so the visual model learns to agree with a reader of the text. Then the relative scores of positive and negative documents for the same query are refined, sharpening the ordering that a reranker is ultimately judged on.

Across eleven datasets from BEIR, RenderRank uses between sixteen point five and thirty-five point five percent fewer input tokens, while reaching an average NDCG at ten of fifty-five point nine six. That outperforms every evaluated text-based baseline below four billion parameters, and some larger models as well.

On four long-document datasets, it averages an NDCG at ten of eighty-eight point two seven, with roughly half the average input token count of the evaluated text rerankers. In that setting its throughput is one point seven times the highest average among the baselines. Long documents are the case where reading each candidate as text is most expensive, so this is the setting the rendering idea was built for, and here the input is roughly halved.

The authors present the results as a demonstration that compressed visual representations can support accurate relevance scoring. For reranking, where every candidate costs tokens, that turns rendered text from a curiosity into a working alternative to reading the tokens one by one.