Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

Lab2Sim. Turning Real-World Experiment Videos into Interactive Virtual Experiments. By Xusen Hei and three others, from the South China University of Technology and the Hong Kong Polytechnic University. Published on the second of November.

A science experiment video shows how an experiment is done, but a learner can only watch it. Turning such a video into a virtual experiment someone can actually operate is harder than it sounds. It means reproducing the apparatus and the state changes on screen, and also enforcing the preconditions that govern each operation. A virtual experiment has to know what may be done, and when, not merely what the bench looked like.

Lab2Sim is a benchmark for exactly that task. It holds two hundred and eighteen science experiment videos, each with specifications grounded in evidence, and eleven thousand eight hundred and eighty-one rubric items in total. Results are judged along four evaluation dimensions, one of which is how faithfully the apparatus is reproduced.

Alongside the benchmark comes a method, EduVideo2Code. It extracts the experimental procedure hierarchically, encodes it as explicit state machines that guide code generation, and refines the resulting pages through verification based on execution.

Existing multimodal models readily produce pages that run, but they struggle to faithfully reconstruct the experimental apparatus. EduVideo2Code improves all four evaluation dimensions consistently, across two backbone models, and apparatus fidelity rises by twenty-four point one five points.

Code that runs is not the same as an experiment that is right. Current models clear the first bar easily and stumble on the second, and Lab2Sim, with its rubric items tied to evidence from each video, is built to tell the two apart.