DecisionRank: On-Policy Distillation for Decision-Model Rerankers

DecisionRank. On-Policy Distillation for Decision-Model Rerankers. By Yanran Wu and twelve others, from Meta and Purdue University. Published on the thirtieth of October.

Decision models handle structured tasks by predicting choices or scores directly, which spares them the latency of autoregressive decoding. A reranker built this way is fast. Teaching it is the hard part, because a generative LLM that already ranks well speaks a different language. The teacher expresses its preferences as a ranked list of document ID sequences. The student decides its ranking from scores read off the first-token position. One produces an ordering as text, the other produces numbers, and transferring knowledge from the first to the second remains challenging.

DecisionRank is a decision model trained to close that gap through what the authors call item-space on-policy distillation, or OPD. The move is to put the student's output into a form the teacher can read. The student's own scores become its rollout, and the teacher responds to that rollout rather than to an ordering of its own.

Concretely, item-space OPD builds the rollout by sorting the student's document scores into a ranked sequence. A generative teacher takes that ranking list as input and gives conditional feedback, which supervises the scores directly. Because the rollout comes from the student itself, the feedback is about the rankings the student actually produces. This on-policy signal is combined with offline ranking supervision during training.

DecisionRank reaches state-of-the-art performance on both general and reasoning-intensive retrieval benchmarks, including TREC DL, BEIR, and BRIGHT. Across four benchmarks, item-space OPD consistently beats alternative training objectives.

Then there is model size. At just one point seven billion parameters, DecisionRank surpasses its twenty-seven billion parameter generative teacher on TREC DL nineteen through twenty-three. At eight billion parameters, it outperforms its generative listwise counterpart, while running twenty-four point five times faster on TREC DL and sixteen point seven times faster on BRIGHT.

A small student overtaking its own teacher is the headline, but the mechanism behind it is a translation. By turning scores into a list the teacher can judge, DecisionRank keeps the speed of a decision model while learning from a generative one, on the rankings it would really produce.