Video2Skill: From Streaming Experience to Reusable Embodied Skills

Video2Skill. From Streaming Experience to Reusable Embodied Skills. By Jianshu Zhang and eight others, from Northwestern University, Carnegie Mellon University, and two other institutions. Published on the twenty-ninth of September. No code repository is linked to it.

Manipulation behaviours vary widely across objects and scenes, yet they share a small set of reusable skills, and planning with those skills helps embodied agents generalize to new tasks. But an agent can only plan with the skills it knows. Recovering skills from observed experience, which is the inverse of planning, builds that knowledge over time, and it also yields skill data for training future agents. Vision-language models describe individual manipulation events well. What had not been measured is whether they can take a continuing stream of events and organize it into reusable skills, which is a different and harder job than describing any one event.

The authors formulate this as Streaming Embodied Skill Discovery, or SESD. A model watches videos in sequence and maintains a persistent skill library, and that library shapes the decisions it makes later. The emphasis is on accumulation. Each new video is judged against everything the model has built up so far, so an early mistake in the library can carry forward into every later decision.

Video2Skill is the benchmark built to measure this ability systematically. It spans two domains, robot tabletop manipulation and human kitchen activity, and it tests three core capabilities. The first is locating manipulation events in time. The second is grouping events that perform the same transformation. The third is deciding, for each event, whether to reuse a skill already in the library or to create a new one. Taken together, the three trace the path from seeing an event to placing it in the library.

Nineteen open-source VLMs are tested. Many of them group events at near-chance level, and scale does not consistently help. Their errors depend on how perception and library updates are coupled. Joint models merge distinct transformations into a single skill. Models that update the library from text descriptions go the other way, and duplicate recurring transformations as if they were new.

Supervised fine-tuning, including the authors' counterfactual library-state rebalancing, called CLaRe, improves grouping. But it exposes a deeper bottleneck. Trained models consolidate the skills they already know, yet rarely expand the library. Their libraries stall below half the reference size, and transformations unseen in training are located in time but almost never given a new skill of their own.

The benchmark ends up pointing past perception. Models can find the events in time. What they struggle with is the judgment that the library they hold is not enough, and recognizing when existing skills are insufficient is what the authors name as the central challenge. Grouping can be improved with training. Knowing when to grow the library is the harder part, and it is the part that stays unsolved.