Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Dream4ACT. A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling. By Xiangyu Zhu and eight others, from the Beijing Humanoid Robot Innovation Center, the Beijing Institute of Technology, and three other institutions. Published on the thirtieth of September. No code repository is linked to it.

Video generation models carry strong spatiotemporal priors, which makes them attractive for embodied modeling of observations and actions together. Leveraging those priors directly is the hard part. Robot actions usually arrive as joint-space vectors, which have no explicit structure in image space, and whose dimensionality and meaning vary from one embodiment to the next. A video model has nothing visual to hold on to. One alternative is to visualize the end effector, but that leaves out the full articulated configuration a robot needs in order to execute.

Dream4ACT is a world model built for joint video-action modeling across embodiments, and its central move is to turn actions into pictures. It introduces a shared visual action interface called action views. An action view renders the target joint configuration from four prescribed virtual cameras, using URDF-based forward kinematics. The result keeps each embodiment's own articulated geometry, yet lives in the same visual space as the observations. That is what lets observation and action sequences share one video autoencoder and one diffusion transformer.

Training uses masked flow-matching. By varying which future sequences are corrupted, a single jointly trained model covers forward dynamics, inverse dynamics, and joint generation of observations and actions together. Then comes the return trip. Predicted action views are images, and a robot needs executable actions, so the authors propose a training-free, URDF-constrained multiview recovery mechanism. It recovers action sequences from the predicted views without any learned decoder specific to an embodiment.

Dream4ACT achieves an average success rate of eighty-eight point nine eight percent on RoboTwin two point zero. On TriWorldBench it reaches an overall score of sixty-five point six six. The two numbers test different things. The success rate is about acting: closing the loop, with the model choosing actions and the robot carrying them out. The benchmark score is about predicting: given actions, generating what several cameras would see next. The authors read these results as support for effective closed-loop manipulation, and for competitive action-conditioned multiview prediction, both delivered through the same visual action interface. One model, one representation, serving both directions.

One representation is doing three jobs at once. Rendering joints as views gives different robots a shared vocabulary. It lets actions borrow the priors of a video model. And it still carries enough geometry to be turned back into commands under URDF constraints, rather than through a decoder trained for each robot.