WorldReward: Reward Modeling for Camera-Conditioned World Models

WorldReward. Reward Modeling for Camera-Conditioned World Models. By Yibin Wang and fifteen others, from Fudan University, Tencent Hunyuan, and four other institutions. Published on the third of September. No code repository is linked to it.

A camera-conditioned world model generates interactive video where a commanded action should produce the expected change in the scene, while appearance, geometry and temporal dynamics all stay coherent. Rewarding that is awkward because existing rewards each see half of it. Geometry-based rewards can estimate whether the trajectory was executed, but cannot judge whether the executed motion looked right. Image-based rewards measure frame quality without capturing action execution or dynamics at all.

The proposal is that a vision-language model supplies a shared reasoning space where actions and their visual outcomes can be related to each other, which is exactly the join the separate rewards are missing. But there is an immediate obstacle the authors are explicit about: judging a complete long video against its full action sequence produces a lengthy, noisy context, and short-lived local evidence that an action was executed can be missed or diluted inside it.

The model therefore works in chunks. Paired videos are decomposed into action-aligned chunks, so each piece of video sits next to the specific commanded action it was supposed to execute rather than next to the whole sequence. Each chunk is organised into structured visual evidence, and chunk-level decisions are then aggregated by voting into separate video-level preferences: one for action consistency, one for visual quality. Keeping those two verdicts apart is deliberate, since they are precisely the two things the earlier rewards each saw only one of, and collapsing them into a single score would reintroduce the problem. Training data comes from a large-scale reasoning-augmented preference dataset, built from structured judgments generated by a frontier model and then refined through tool-based agent auditing and targeted human review, which is a three-stage pipeline rather than a single generation pass.

The authors also build a human-annotated benchmark measuring how well a reward model agrees with human preferences across action consistency, appearance quality and motion quality. Their model achieves the highest agreement on all three, exceeding GPT-5.5 by three point four two, one point four five, and three point five six percentage points. Used for reinforcement learning post-training of a world model, it consistently improves both action execution and visual quality across short to long horizons, which is the check that a reward model is measuring something real rather than something optimisable.

The structural point is that a reward for interactive video has to answer two different questions at once, and that chunking is what keeps a short piece of evidence from being averaged away inside a long one.

WorldReward. Reward Modeling for Camera-Conditioned World Models. By Yibin Wang and fifteen others, from Fudan University, Tencent Hunyuan, and four other institutions. Published on the third of September. This is the long version: the whole paper rather than the abstract.

Video world models simulate how a visual environment evolves in response to your controls, and the field has progressed from predicting future observations to explicit camera-trajectory control and on to real-time long-horizon interaction driven by keyboard and mouse. A generated video is useful only if it faithfully executes the commanded controls while keeping geometry, appearance and temporal dynamics coherent over the whole horizon. Which makes reward models central here, because they determine both how these systems are evaluated and, increasingly, how they are optimized.

The paper names three challenges, and taking them one at a time pays off because the design answers each in turn. The first is coupled requirements. The same visual change can indicate correct or incorrect execution depending on what was commanded, and two videos with similar trajectory accuracy can still differ in appearance or dynamics. So action consistency and visual quality should be judged from a shared interpretation of the video, rather than as two separate outcomes computed by two separate systems. The second is localized evidence: a forward or turning command manifests within a few frames of a long sequence, so a judge has to find that short-lived transition without being swamped by everything else. The third is long-horizon attribution, because failures accumulate as generation proceeds, so a local motion error has to be attributable to the action segment that produced it.

Prior work did combine the two signals for post-training, which the authors acknowledge is effective, but it leaves action execution and visual quality assessed by heterogeneous, decoupled systems. General video preference models capture perceptual quality without verifying whether the commanded actions were followed. And direct judging with a vision-language model over all frames and the complete action sequence runs straight into the second challenge: a lengthy noisy context in which short-lived local evidence gets missed or diluted. The proposal is a pairwise preference reward model that unifies both judgments in one vision-language reasoning space, and that resolves the context problem by chunking. Paired videos are decomposed into action-aligned chunks, so each piece of video sits beside the specific command it was meant to execute. Each chunk is organised into structured visual evidence, letting the model evaluate execution and visual quality together on the same material. Chunk-level decisions are then aggregated by voting into two separate video-level preferences.

Training data comes from a large-scale reasoning-augmented preference dataset, built from structured judgments generated by a frontier model and then refined through multi-turn tool-based agent auditing and targeted human review, so three stages rather than one generation pass. The authors also build a human-annotated benchmark measuring how well a reward model agrees with human preferences across action consistency, appearance quality and motion quality. Their model achieves the highest agreement on all three, exceeding GPT-5.5 by three point four two, one point four five and three point five six percentage points respectively. Used for reinforcement learning post-training of a world model, it consistently improves both action execution and visual quality across short to long horizons. The human study is blinded: two hundred video pairs from the long-horizon combined-action subset, sharing source image, prompt and action trajectory, with model identities hidden, left-right order randomized, and three independent annotators per pair.

The comparison that most clarifies the contribution is the qualitative one. The prior combined approach carries an image-based reward that scores sampled frames independently, without directly assessing whether colour and texture stay consistent over time, and the post-trained model here maintains a more stable appearance across the trajectory as a result. That is presented as motivating temporal visual-quality feedback during post-training, which is a reasonable reading and also a reminder of the scope: everything is camera-conditioned world models with a specific backbone, the reward is trained on judgments from a frontier model whose biases it will inherit, and agreement with human preference is the target rather than any external notion of physical correctness.

The structural lesson is that a reward for interactive video has to answer two questions at once, and that where you place the split determines what you can see. Judge jointly and report separately, and chunk the evidence so a two-second turn is not averaged into a two-minute sequence.