SimForcing: Distilling Simulation Motion Priors into Real-Domain Robot World Models

SimForcing. Distilling Simulation Motion Priors into Real-Domain Robot World Models. By Xiaodong Wang and six others, from Peking University. Published on the fifth of October. Its code repository is worked out: about forty source files, with a readme and a licence.

An action-conditioned robot world model has two jobs at once. It must respond precisely to the robot trajectory it is given, and it must keep the visual dynamics of the predicted video realistic. Learning both from heterogeneous robot videos is hard. Simulation looks like an obvious help, because it offers structured motion supervision. But two things get in the way. Simulated footage does not look like real footage, and those appearance differences hinder direct transfer. And simulation predictions can simply be wrong, in which case they misguide the real video being generated.

SimForcing is a simulation-guided framework that uses simulation in two roles at once. In the first role, simulation is a source of motion knowledge that can be transferred. In the second, it is a controllable reference that the model can consult while predicting. Each role is designed around one of the two obstacles: appearance on one side, and inaccurate simulation on the other.

For the first role, motion knowledge moves from a simulation teacher through latent-motion distillation. The student aligns temporal changes in latent space, which lets it internalize motion priors while mitigating the influence of appearance differences. For the second role, the authors introduce multi-block simulation conditioning with condition dropout. The model exploits predicted simulation trajectories, but without relying too heavily on their accuracy.

A simulation-conditioning classifier-free guidance scheme ties the two ideas together. It balances predictions that rest on internalized motion knowledge against predictions additionally guided by simulation latents. The student is trained jointly to generate both the simulation conditions and the real-domain videos. That matters at inference: no additional world model is needed to supply the simulation side.

On Bridge, SimForcing reports the best PSNR, SSIM, LPIPS and FVD among the compared methods, and it does so without external embodied pretraining. Evaluation on InternData-A1 further supports its use across robot datasets. There is also a downstream test. Using the trained world model to initialize a vision-language-action model improves success on LIBERO, which the authors read as a sign of its utility for policy learning.

Simulation here is neither trusted outright nor thrown away. Its motion is absorbed through latents so that appearance stays out of the way, and its predictions are consulted with dropout so that their errors cannot dominate. Guidance then weighs the two, inside a single student that carries everything it needs at inference time.