It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

It Takes Two to Match. Co-Evolving Generative Retriever with Reinforcement Learning. By Runpeng Dai and three others, from the University of North Carolina at Chapel Hill and Apple. Published on the first of September. No code repository is linked to it.

In modern search and advertising systems, retrieval comes first. It selects a candidate set from a very large universe of items, and everything downstream, ranking and auction, works from that set. Language models are increasingly used to improve this stage, through query expansion, data synthesis, and training on retrieval feedback. But the generative part usually only augments the query. The final matching is still handed off to a retriever further down the line.

CoGR takes a different route. It trains language models to construct the retrieval representations themselves, on both sides: one generator for queries, one for items. Each generator produces a compact set of keywords, and those keywords are matched directly through an inverted index. That keeps the approach compatible with the keyword-based retrieval infrastructure already in place.

Training has two stages. Supervised fine-tuning comes first and establishes a keyword space the two sides share. Then comes co-evolving reinforcement learning. The query generator and the item generator are optimized in turn with GRPO, each against a frozen index built by the other side. While one side learns, the other stays put, so each generator is always aiming at a fixed target. Both chase the same objective, the F one score of retrieving the right items for a query.

The two sides earn that reward differently, because they stand in different places. The query side issues the search, so it receives the retrieval F one score directly. The item side never searches at all. It is only ever found. So it receives a counterfactual marginal reward instead: the change in the query side's F one score that its own generated keywords caused. An item's keywords are credited by how much they help queries find it.

CoGR is compared against ten representative baselines, spanning sparse, dense, and generative retrieval. It achieves the best performance on two datasets. On an internal APP Marketplace dataset, it improves the F one score over the strongest baseline by ten point nine percent. On the public WANDS benchmark, the improvement is thirty-six point one percent. Further analysis shows that the co-evolution is stable, and that the keyword spaces of queries and items grow increasingly aligned as training proceeds.

The title's phrase, it takes two to match, carries the design. Neither side can learn its keywords in isolation, because a match only happens when both choose the same words. Training them in alternation, each against the other's fixed index, and rewarding the item side by its effect on the query side, turns two separate generators into partners converging on a shared vocabulary.