DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
Zhongxi Chen, Yi Yang, Xinyi Chen, Mingzhe Ye, Yaping Li, Zhihong Liu, Congsheng Xu, Haonan Wang and 1 others
Sharpa Robotics · SJTU · CUHK · FDU
Published November 10, 2026 · on the feed November 11, 2026 · ▲ 24 · arXiv 2611.11166
The abstract
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
The Brief
2:20A short podcast summary of the paper. Written and voiced by AI.
Transcript
DexForesight. Grounding Dexterous Policies with Implicit Spatial Foresight. By Zhongxi Chen and eight others, from Sharpa Robotics, Shanghai Jiao Tong University, and two other institutions. Published on the tenth of November.
Dexterous manipulation needs rich spatial foresight, about the surrounding scene and about the robot hand itself. Existing world-action models tend to acquire that predictive knowledge through explicit future prediction: the future is something they produce, as an output, alongside the actions. The question this work asks is whether a policy needs to produce that future at all, or whether it is enough for the policy to carry knowledge of it.
DexForesight takes the second route and learns the foresight implicitly. Dynamics-aware world representations are injected into the latent space of a dexterous policy, so the policy carries predictive knowledge without having to output a predicted future. The foresight lives where the policy does its thinking, in its latents, and the world model that supplies it is needed only while the policy is being trained.
Training starts with an action-conditioned spatial world model, learned from offline scene depth, kinematic hand geometry and demonstration trajectories. This world model predicts how points on the hand move in 3D and how visual features change. Between them, those predictions capture the articulated dynamics of the hand and changes in the scene under the demonstrated actions, including the motion of fingers the cameras cannot see.
Next, the world model is frozen. The policy's queries are aligned with the world model's predictive queries, which transfers the learned spatial dynamics into the policy's latents, and those latents condition action generation.
At deployment the world model is dropped entirely. Only the policy remains, running on RGB observations, language and robot state, with no depth input and no explicit future prediction.
Across eleven DexJoCo simulation tasks, DexForesight lifts the base policy's overall success rate from forty-nine point four percent to sixty point five percent. On three dexterous manipulation tasks with a real robot, it reaches fifty-eight point three percent success, against fifty point zero percent for the strongest baseline.
Depth and hand geometry are used once, during training, and then left behind. What survives into deployment is their effect on the policy's internal representation, shaped to agree with a model that knew how the scene and the hidden fingers would move. Foresight becomes a property of the latent space rather than an output the robot must compute at run time.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A hand with hidden fingers, and a policy that only sees the present. That is the wall.
You are teaching a robot hand to grasp and turn objects. Its fingers must move around surfaces while some of them are hidden from the cameras, so seeing the current scene is not enough.
The road ahead
- 1 · The problem ~6 min A robot hand needs to anticipate how its fingers and the scene will move, but imagining the future at every step is costly.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min A training-only world model learns how hand and scene change, and the policy's own latents are taught to carry that knowledge.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train a world model with training-only signals, freeze it, align the policy's queries with it, condition actions on them, then deploy the policy alone.
- 6 · The experiment ~6 min You’ll guess: On 11 DexJoCo simulation tasks the base policy succeeds 49.4% of the time overall. What overall success rate does DexForesight reach?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.