LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures

LabInstruct. Benchmarking Situated Instructional Video Generation for Lab Procedures. By Yuming Fu and seven others, from Central South University, the National University of Singapore, and three other institutions. Published on the twenty-ninth of September. Its code repository is public and real: about twenty-five source files.

Self-driving laboratories aim to automate the whole experimental loop, from scientific decision-making through to physical execution. Ideally, plans written by AI would be carried out directly by robots. But reliable automation is still hard in complex, open-world labs. Experiments there involve fine-grained manipulation, long procedures, and a great deal of variation from one task and one setup to the next. So humans remain an important execution interface, standing between the plans an AI produces and the physical experiment. That creates a need for guidance aimed at people: clear, effective, and easy to follow.

Lab procedures are inherently visual, spatial and dynamic, which makes video a natural medium for that guidance. It can show how apparatus is configured, what manipulation is performed, what depends on what in time, and how states change. Recent video generation can now synthesize an experimental demonstration directly from an image of the starting workspace and an instruction in natural language. Whether such models can reliably communicate real lab procedures, though, has not been systematically studied.

LabInstruct is a benchmark for situated instructional video generation in real laboratories. Situated here means the video has to start from the actual workspace in the image. And the question is not only whether a generated video looks real, but whether it gets the procedure right.

The benchmark contains two hundred and four tasks, drawn from five scientific disciplines. Each task comes with a real reference execution, so there is a recording of the procedure done properly. Each also carries structured annotations of the objects, actions, contacts and state transitions involved, which give the evaluation something concrete to check against.

Eight frontier image-to-video models were evaluated. The main finding is that generations which look visually plausible frequently remain procedurally incorrect. The authors describe a substantial gap between visual realism and the reliability that experimental instruction requires.

Looking right and being right are different properties, and for someone following a video at the bench, only the second one counts. With real reference executions and annotated steps, this benchmark gives a way to tell the two apart.