Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation
Zihan Wang, Zhen Wu, Pieter Abbeel, Rocky Duan, Jitendra Malik, Carmelo Sferrazza, C. Karen Liu, Guanya Shi and 1 others
Amazon FAR · UC Berkeley · Carnegie Mellon University · Stanford
Published September 29, 2026 · on the feed September 30, 2026 · arXiv 2609.38172
The abstract
Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach. We propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set. PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. Our contact-anchored real-to-sim pipeline then reconstructs both human and object motions, retargeting this imperfect video data into physically plausible trajectories. The intra-class variability across these counterfactual videos lets us train a single policy that generalizes to unseen objects within each category. We demonstrate the full pipeline by deploying this policy on a real robot without any real-world fine-tuning. Using only onboard depth observations, our humanoid picks up, carries, and drops objects, including boxes, barrels, bins, and balls, across novel instances, sizes, and initial configurations.
The Brief
2:04A short podcast summary of the paper. Written and voiced by AI.
Transcript
Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation. By Zihan Wang and eight others, from Amazon FAR, UC Berkeley, and two other institutions. Published on the twenty-ninth of September. No code repository is linked to it.
One promising route to generalist robots is teaching humanoids loco-manipulation, such as carrying all sorts of objects, by having them imitate what they see in video. The obstacle is the video. Scaling this up needs many diverse, high-quality interaction clips, ones that clearly show a person's whole body and leave their contact with the object unoccluded. Collecting those is a practical barrier.
PRISM gets around the barrier by multiplying a few real videos into many. It is a real-to-sim-to-real framework. From a handful of exemplar real videos, it generates hundreds of diverse counterfactual videos of a human interacting with an object, using video-to-video generation. Each one shows a variation on the original that was never actually filmed.
Generated video is imperfect, so it is not used raw. A contact-anchored real-to-sim pipeline reconstructs both the human's motion and the object's motion from each clip, and retargets that imperfect data into trajectories that are physically plausible. The counterfactual videos vary within each object class, and that variability is what lets a single policy, trained on all of them, generalise to objects of the same category it has never seen.
The full pipeline is demonstrated on a real robot, with the policy deployed without any fine-tuning in the real world. Using only onboard depth observations, the humanoid picks up, carries and drops a range of objects: boxes, barrels, bins and balls. It does so across new instances of those objects, different sizes, and different starting configurations. The abstract reports these as demonstrations rather than as a table of rates.
Here, generated video stands in for real footage nobody had to film. A few real clips become hundreds of counterfactual ones, those become physically plausible motion, and the variety in that motion becomes a policy that copes with boxes and barrels it was never shown.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a shortage of the right kind of video.
You want a humanoid robot to pick up, carry and drop all sorts of objects by imitating people in videos. For that you need clips that show the whole body and the hands on the object, unoccluded.
The road ahead
- 1 · The problem ~6 min Robots can learn to carry objects by imitating people in videos, but clean full-body interaction clips are rare and costly to find.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Remake a few real clips with other objects using video generation, then use human-object contact to turn them into plausible robot training data.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Seed with a few real clips, generate variants, reconstruct and retarget them with contact, train one policy, and deploy it on a robot.
- 6 · The experiment ~6 min You’ll guess: Trained only on trajectories from generated videos, did the humanoid's policy need real-world fine-tuning before it could handle new objects?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.