PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

PDRVLA. A Vision-Language-Action Model with Physical Dynamics Representations. By Hanqing Wang and fourteen others, from Peking University and JD.com. Published on the twenty-seventh of October.

Vision-language-action models turn camera views and language instructions into robot actions, but their grasp of physical dynamics is limited. Video world models do capture how a scene evolves. The catch is that generating future video puts high latency inside the control loop.

PDRVLA tries to have the physics without the video. A lightweight Physical Dynamics Module pulls physical tokens out of the vision-language features, and those tokens condition the generation of actions. No video is generated online. The bet is that a model does not need to render the future in pixels to act on an understanding of it. A compact summary of the dynamics, drawn from features it already computes, can carry what the action generator needs.

The module learns by directly predicting future features of three kinds: RGB, depth, and optical flow. A frozen, pretrained video world model adds auxiliary supervision on top, so the knowledge of a full video model reaches the module during training without the module having to generate video. All the training data is publicly available, and it is expanded in three ways: synthesis that turns human data into robot data, subtask annotation, and embodied chain-of-thought annotation, known as ECoT.

Four simulation benchmarks are reported. PDRVLA reaches ninety-nine point three percent success on LIBERO and eighty-nine point nine percent on LIBERO-Plus.

On RoboTwin two point zero it reaches ninety-three point six percent, and on SimplerEnv eighty-one point five percent. Ablations show that world-model supervision improves future prediction, and they measure how subtask conditioning, future object-pose supervision and ECoT training each affect action success.

The design keeps video generation out of the control loop. The world model serves as a source of supervision during training, and what conditions the actions is a lightweight set of physical tokens.

PDRVLA. A Vision-Language-Action Model with Physical Dynamics Representations. By Hanqing Wang and fourteen others, from Peking University and JD.com. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

A vision-language-action model looks at camera images, reads an instruction, and outputs robot motion. What it understands about physics is thin. The authors cite an analysis reporting that several such policies encode object properties weakly and do not reliably adapt their motions when an object's mass changes. Manipulation is a problem about change: grasping, moving and placing alter where things are and how they relate. Anticipating how the scene will evolve could help choose the next action.

Video world models do learn that kind of change, by predicting how scenes unfold over time. The catch is speed. Generating a future video inside the control loop adds latency to every action. PDRVLA, from JD.com and Peking University, tries to keep the knowledge and drop the video.

Two pieces of background make the design readable. The first is the action expert. Rather than emitting one command, the policy produces a chunk of eight commands at once, using flow matching: it starts from Gaussian noise and follows a predicted velocity toward a clean action chunk. At inference this takes four Euler steps, and the whole chunk runs before the policy replans from a new observation.

The second is where visual information lives inside a vision-language model. The backbone here is Qwen3.5-4B, with thirty-two decoder blocks. The paper cites layer-wise studies finding stronger perceptual representations at intermediate layers, and reasons that later layers may suppress visual information useful for predicting physical change. That motivates tapping the model partway through rather than only at the top.

The Physical Dynamics Module reads features from block eight of the backbone. Fifty learnable queries pass through four small Transformer blocks: in each, they cross-attend to those intermediate features, exchange information through self-attention, and go through a feed-forward layer. A projection then turns them into fifty physical tokens at the backbone's width. The module adds about fifty-seven million parameters, one point four percent of the backbone. The action expert, a Diffusion Transformer of about a hundred and fifty million parameters, conditions on the final-layer features and the physical tokens joined into one sequence, plus the robot's current state.

Nothing in that wiring makes the tokens physical. Training does. Each example pairs the current observations with a five-frame clip spanning the following thirty action steps. Off-the-shelf estimators turn the RGB into depth and optical-flow videos, and a frozen Wan video autoencoder compresses all three into latent features. A small head reads the physical tokens and regresses pooled, standardized versions of those future features. That is direct prediction.

On top sits the world-model term. A frozen video model, LingBot-Video-Dense-1.3B, receives the RGB latent clip with its future part mixed with noise, takes the physical tokens as conditioning, and must predict the denoising velocity. Two new trainable heads extend this to depth and flow. The video model itself is never updated, but gradients pass through it into the module, so the tokens are pushed to carry whatever helps a video predictor anticipate the future. The direct-prediction weight ramps from zero to one half over the first two thousand steps.

Then the video model is set aside. Joint policy training updates the backbone, the module, the action expert and the direct head together, keeping direct future prediction alive at a tenth of the action loss's weight. One detail matters: the intermediate features feeding the module are detached, so the action and prediction losses shape the module without reaching back into the backbone through that path.

On the four simulation benchmarks PDRVLA posts the top average in each table: ninety-nine point three percent on LIBERO, eighty-nine point nine on LIBERO-Plus, ninety-three point six on RoboTwin 2.0, and eighty-one point five on SimplerEnv. The margins are small: two tenths of a point over Qwen-RobotManip on LIBERO, four tenths over two tied rivals on RoboTwin. It loses individual tasks too, such as carrot to plate, where it scores sixty-four point six against eighty-five point four for InternVLA-A1.5.

The test closest to the thesis is whether the world model helps. Two runs with equal budgets, one with direct prediction only and one adding world-model supervision, are scored on eight hundred and twenty-eight held-out RoboCOIN clips. Mean prediction error falls by one point three two percent. RGB improves by about four percent and depth by about two, while the flow improvement's interval includes zero.

The auxiliary studies are mixed. Training on embodied chain-of-thought lifts the LIBERO Long suite from ninety-two point eight percent to ninety-six, yet generating that reasoning at inference adds no consistent gain. Self-generated subtasks help when refreshed every one or eight chunks, but refreshing every thirty-two drops below the baseline.

There is no limitations section. The conclusion names only future work: predictive targets such as object-centric geometry, contact states and material properties.

The evidence leaves a gap. The world-model comparison measures future-feature prediction, not action success, and no reported experiment removes the Physical Dynamics Module and measures what the policy loses. So the paper shows the tokens predict the future somewhat better with world-model help, and that the full system scores well, but not that the tokens are why. All four benchmarks are simulated. Baseline numbers come from cited papers and repositories under their own protocols, the design comparisons are single-seed, and the policy ablations report no intervals.

The idea to carry away is training-time foresight. Use an expensive video world model as a teacher whose gradients sculpt a cheap set of fifty tokens, then deploy without it. Read the result carefully, though: what is shown is better prediction and strong benchmark scores, not yet a measured link between the two.