DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai and 1 others
National University of Singapore · Kling AI · HKUST
Published October 8, 2026 · on the feed October 12, 2026 · ▲ 148 · arXiv 2610.12572
The abstract
Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.
The Brief
2:27A short podcast summary of the paper. Written and voiced by AI.
Transcript
DiffusionART. Efficient, Stable and Scalable Reinforcement of Diffusion Models. By Zhenglin Wan and eight others, from the National University of Singapore, Kling AI, and one other institution. Published on the eighth of October. Its code repository is worked out: about two hundred source files, with a readme.
Online reinforcement learning fine-tunes diffusion models against black-box rewards, and the existing methods fall into two camps with opposite weaknesses. Reverse-process methods, such as Flow-GRPO, are held back in efficiency by poor credit assignment. Forward-process methods, such as DiffusionNFT, are efficient, but unstable unless off-policy training with an exponential moving average is carefully tuned.
The authors start from a diagnosis. Objectives in the style of NFT contain a hidden consistency force between the forward and reverse processes, and joint training on positives and negatives cancels it only in part. The moving average, they show, stabilizes training through restoration, not through its off-policy rollouts. DiffusionART, short for Diffusion Reinforcement with Aggregation-First Reward Tilting, is built on both observations.
First, it aggregates each prompt's rewards into one shared direction that improves reward, applied to reverse-process states. That removes the consistency force by construction. Second, it adds a second update, masked by a trust region, which gives restoration that adapts to drift, with no moving average to tune. It needs no differentiable reward model, it stays effective with fewer samples per prompt, and a variant called DiffusionART-Pairwise supports pairwise reward models.
Across image and video generation, DiffusionART trains efficiently and stably without tuning per task. The headline case is HunyuanVideo-13B, with visual and motion quality rewards weighted equally. There, DiffusionART reaches the late-stage mean reward of DiffusionNFT using twelve point zero two times fewer GPU hours, and that of Flow-GRPO-Fast using eleven point seven two times fewer.
Along the way it achieves mean training rewards that are twenty-eight point eight percent higher than DiffusionNFT, and fifty-three percent higher than Flow-GRPO-Fast.
Explaining why the stable method needed its moving average turned out to be the route to dropping it. Once restoration is understood as the real job, a targeted update can do that job directly, and the cancelling force can be designed out rather than tuned around.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Two ways to train with rewards: one slow, one fragile. You need neither flaw.
You fine-tune an image and video generator with reinforcement learning against black-box rewards. One family of methods updates sampled denoising steps, but credit assignment is poor, so it learns slowly.
The road ahead
- 1 · The problem ~6 min Fine-tuning a diffusion generator with black-box rewards forces a choice between a slow family of methods and a fast but fragile one.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Pool each prompt's rewards into one shared improving direction for visited states, which removes a hidden force, and add a masked second update for restoration.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each iteration rolls out samples, pools rewards into a shared direction, regresses visited states toward a frozen target, then runs a masked second update.
- 6 · The experiment ~6 min You’ll guess: On HunyuanVideo-13B, how many times fewer GPU hours did DiffusionART need to reach DiffusionNFT's late-stage mean reward?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.