Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

Many Right Answers, One Training Example. Removing Responder Noise from GRPO for LLM Recommenders. By Xin Li and four others, from Nanyang Technological University. Published on the thirteenth of October. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.

Picture a recommendation request on Reddit. It draws several replies, and each one names different titles. Now train an LLM recommender on threads like that with reinforcement learning. Which of those replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example. The consequence, by the authors' analysis, is that the advantage mostly records which reply happened to be drawn, eighty-five percent of its variance, rather than how good the list actually is. The same list could be praised against one reply and penalized against another, and the training signal ends up describing the responder more than the recommender. That is the noise in the title.

The argument is that reinforcement learning should train on the request, not on each answer. A request with many good replies is one training example with many right answers, not many separate examples that each have one. The method that follows from it is called ReMaRk.

ReMaRk scores every generated list once, against the union of the titles named across all of the replies. It also samples each request equally, instead of letting a request with many replies count many times. One request, one reward, built from everyone's answers together. A side effect is that the training becomes cheaper.

The comparison spans seven backbones and three datasets. ReMaRk reaches the validation recall that per-reply training gets after two thousand steps in only twenty-five to fifty-three percent of those steps, and it leads at that budget. When both are trained until neither improves any further, ReMaRk is never significantly behind Rank-GRPO, and in four of the seven pairs it reaches Rank-GRPO's best result using fewer GPU-hours.

The same reward also helps on ProtoQA, a setting where about one hundred people answer each question. That is a long way from recommendation threads, and it shares the one feature that matters: many valid human answers to a single prompt.

Recommendations were the starting point, but the ProtoQA result is what lets the authors state a general rule. Whenever reinforcement learning faces several valid human answers to one prompt, reward them jointly, rather than letting a random draw of one of them decide what the model is told.