ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
Hongxing Li, Dingming Li, Yixin Li, Yong Du, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang and 1 others
Zhejiang University
Published October 8, 2026 · on the feed October 9, 2026 · ▲ 15 · arXiv 2610.12403
The abstract
Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.
The Brief
2:02A short podcast summary of the paper. Written and voiced by AI.
Transcript
ViSkill. Reinforcing VLM Agents with Evolving Visual-Native Skills. By Hongxing Li and eight others, from Zhejiang University. Published on the eighth of October.
Agents that carry a library of skills learn more efficiently, because successful trajectories get distilled into strategies they can reuse. Most of these approaches are text-centric, though. Spatial layouts and the correspondence between actions and states are flattened into language, and the geometric structure that mattered is lost on the way. Some recent work has started to bring visual evidence in, but it builds and updates skills separately from policy optimization. Whether the two could improve each other has been left largely unexplored.
ViSkill is a visual-native skill learning framework. Successful interactions are encoded as composite visual skill cards, which VLM agents can access directly rather than through a textual description. A skill, in other words, stays in the same form as the observations the agent is already looking at, instead of being translated into words and back again. Whatever geometry the interaction depended on is kept on the card.
Retrieved skills do two jobs. They guide inference, and they shape the reward. Meanwhile, successful trajectories are distilled back into the library. That closes a feedback loop: a growing library makes the policy better, and a better policy feeds the library with more successes. An optional cold-start mechanism speeds up the early stage of learning, before the library has much in it.
Evaluation covers three environments: Sokoban, FrozenLake, and PrimitiveSkill. ViSkill reaches an overall success rate of zero point eight nine, and zero point nine one with cold-start initialization. It outperforms every proprietary and open-source baseline evaluated, and it converges faster than standard PPO. The cold-start gain is small at the end, two hundredths, which fits its stated purpose: it is there to speed up early learning rather than to raise the final ceiling.
Two ideas carry the work. Skills are kept in the visual form the agent actually perceives, so geometry survives. And the library is not a side product assembled after training but part of the training loop itself, where skills and policy improve together.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free