Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards. By Yuanhao Ban and five others, from Arena Intelligence and UCLA. Published on the second of October. No code repository is linked to it.

Text-to-image models now produce remarkable visual quality, yet improving them further through post-training is hard. The obstacle the authors name is the reward. No single reward signal captures the full range of what people prefer in an image. A model optimized against one narrow signal learns to satisfy that signal, and everything the signal leaves out is left to chance. For open-domain generation, where prompts can ask for almost anything, that gap is wide.

The answer proposed here is a simple post-training recipe built on composition. Instead of searching for one reward that covers everything, the authors combine complementary reward signals, each covering ground the other misses. The recipe is meant to be simple and effective rather than elaborate, and its central contribution is not any one reward but the way the rewards are put together.

The reward system has two main components. The first is a preference reward, trained on large-scale human preference data with a Bradley-Terry objective. Its job is to capture overall human aesthetic and perceptual preference: the broad sense of which image people would rather look at. The second is a set of rubric-based rewards. These explicitly evaluate prompt faithfulness and other desirable properties, and they also provide safeguards against reward hacking, so the model cannot drift into images that please one scorer while ignoring the request.

That leaves a real difficulty. These are heterogeneous signals, and they have to be combined into one training objective. The obvious move is a weighted average, and the authors show that a naive weighted average leads to suboptimal optimization behavior. They propose instead a simple reward composition strategy that more effectively balances preference optimization against rubric satisfaction, so that pushing on one does not quietly give up the other.

The results are reported on the Arena text-to-image leaderboard. An RL-trained Flux2dev reaches an Elo rating sixty-nine points above its base model. A post-trained Ideogram model goes further: it surpasses every open-source model on the leaderboard, reaching an Elo of one thousand two hundred and twenty-three point five. The authors note that these state-of-the-art claims rest on a leaderboard snapshot taken in early September.

For reproducibility they release Arena-T2I-Training, a one-thousand-example subset of their training data. It recovers some of the gains of full-scale training, and is offered as a resource for future work on post-training text-to-image models.

The lesson the authors draw is about rewards rather than models. Effective rewards for training frontier generative models need two properties at once: broad coverage of what users intend, and robustness to exploitation once optimization starts pushing on them. A preference reward supplies the first, rubrics guard the second, and the composition rule decides whether the two actually cooperate. On their account, getting that rule right is what separates a recipe that works from a weighted average that stalls.