Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
Kerui Ren, Yingxiang Xu, Kaiwen Song, Linning Xu, Bo Dai, Mulin Yu, Tao Lu
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Zhejiang University · University of Science and Technology of China · The Chinese University of Hong Kong · The University of Hong Kong
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 6 · arXiv 2609.37089
The abstract
Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.
The Brief
2:37A short podcast summary of the paper. Written and voiced by AI.
Transcript
Real2Gym. Building Gyms from Videos, Bringing Skills to Robots. By Kerui Ren and six others, from the Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and four other institutions. Published on the twenty-ninth of September. Its code repository is public but slight: four source files.
Real-world videos provide rich demonstrations of manipulation. Turning them into robot skills that can be reused takes three things the videos do not supply on their own: environments that are visually aligned with the footage, physical interactions that can actually be executed, and some mechanism for learning from experience. A video shows what happened. It does not give a robot a place to try it, a guarantee that the motion is physically possible, or a way to get better after a failed attempt.
Real2Gym is an agentic Real2Sim2Real framework. It turns demonstrations by humans and by robots into interactive simulation gyms, and then brings the skills acquired in simulation back out to physical robots. The name traces the route: from the real world into simulation, and from simulation back to the real world. Each of the three missing pieces is handled somewhere along that route.
The Real2Sim module reconstructs editable scenes and aligns objects and cameras with the input. It validates the demonstrated or retargeted actions by executing them in native physics, and generates task-conditioned variations, with checks that the actions remain feasible. The gym, in other words, is built to match the footage and to obey physics.
Inside those environments, the agent writes executable code for each stage of a manipulation and observes the outcome. Successes and failures alike are distilled into reusable task procedures, object-relative motions and recovery strategies. A shared interface for perception and control then lets those skills guide later execution, in simulation and on real robots. Motions are adapted to current observations, and the underlying model weights are never updated.
The evaluations report high-fidelity reconstruction of simulation environments. Across those environments, Real2Gym outperforms GPT-6 Astra Direct Mode by sixteen point seven percent in success rate, while using approximately seventy-four point nine percent fewer policy-execution tokens. On a real Franka robot, across four tasks, it exceeds the same baseline by thirty-three point three percent in physical execution success rate.
What the system accumulates is not a retrained network but a set of procedures, motions and recoveries, written down and reused. A video becomes a gym, the gym becomes experience, and that experience reaches the physical robot without a single weight being changed.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A video shows what happened. It does not give a robot a place to try.
You have videos of people and robots handling objects, and you want a robot to learn those skills. Practising by trial and error on real hardware is slow, risky and needs constant supervision.
The road ahead
- 1 · The problem ~7 min Robots could learn skills from videos by practising, but real hardware makes practice costly and automatically built simulated scenes are too inaccurate to practise in.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Rebuild the video's scene in simulation, prove it works under physics, let an agent learn reusable skills there by coding, then carry them to a real robot.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Rebuild an aligned scene, verify the demonstration under physics, make checked variations, distill the agent's attempts into skills, then run them on a real robot.
- 6 · The experiment ~6 min You’ll guess: On a real Franka robot across four tasks, by how much did Real2Gym exceed GPT-6 Astra Direct Mode in physical execution success rate?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.