EVISKILL: Grounding Skill Evolution in Replayable Evidence
Yan Zhou, Yili Wang, Yiwei Dai, Qinggang Zhang, Xin Wang
Jilin University
Published October 4, 2026 · on the feed October 7, 2026 · ▲ 52 · arXiv 2610.05030
The abstract
Continual skill evolution enables LLM agents to accumulate and refine reusable procedural knowledge from interaction experience without updating model parameters. Its effectiveness depends on determining not only what to change, but also why a change is justified and when it should become persistent guidance. However, existing experience-driven methods can lose the behavioral evidence and task contexts supporting edits. Moreover, a global validation outcome provides an incomplete judgment of its constituent changes: locally supported corrections may be discarded with a rejected revision, while evidence may require further experience to inform useful updates. To this end, we introduce EVISKILL, an evidence-driven framework that organizes execution observations into Replayable Evidence Cards and synthesizes edits with explicit links to their supporting contexts. Targeted replay verifies these edits through re-execution and provides feedback for correction. Across epochs, EVISKILL preserves evidence and provisionally retains supported edits for further refinement, while global validation governs their incorporation into the final skill. Experiments on three interactive benchmarks across six LLM backbones demonstrate the effectiveness of this approach.
The Brief
1:57A short podcast summary of the paper. Written and voiced by AI.
Transcript
EVISKILL. Grounding Skill Evolution in Replayable Evidence. By Yan Zhou and four others, from Jilin University. Published on the fourth of October. Its code repository is public and real: about forty source files.
Continual skill evolution lets language model agents build up and refine reusable procedural knowledge from their own interaction experience, with no change to the model's parameters. Doing it well means deciding three things: what to change in a skill, why the change is justified, and when it should become persistent guidance. Existing experience-driven methods can lose track of the second. The behavioural evidence and task contexts that supported an edit go missing. There is a further problem with validation. A single global outcome is an incomplete judgment on the many changes inside a revision. A correction with good local support can be thrown out along with a revision that was rejected as a whole, and some evidence needs more experience before it can produce a useful update at all.
EVISKILL is driven by evidence. It organizes what it observes during execution into Replayable Evidence Cards, and it synthesizes each edit with explicit links back to the contexts that support it. Every change carries its reasons with it.
Those links make edits testable. Targeted replay re-executes the relevant context to verify an edit, and the result feeds back as correction. Over successive epochs, EVISKILL keeps the evidence, and provisionally retains edits that have support so they can be refined further, rather than losing them to a single verdict. Global validation still has the final say, but its role is narrower: it governs which edits are incorporated into the finished skill.
Experiments cover three interactive benchmarks and six language model backbones, and the authors report that they demonstrate the approach is effective.
The design separates two judgments that a single global check had merged. Whether an individual edit is supported is settled locally, by replaying its evidence. Whether the skill as a whole should change is settled globally. Good corrections no longer have to share the fate of the revision they arrived in.
Paper Quest · 7 chapters · about 38 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An agent that learns from experience can also forget why it learned.
You run an agent that keeps a written skill file and improves it from its own runs, without touching the model's weights. After each round, edits are proposed from what went right and wrong.
The road ahead
- 1 · The problem ~7 min An agent can improve its own written skill file from experience, but edits lose their reasons and good fixes get thrown out with bad revisions.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Tie each skill edit to replayable evidence from its runs, test it by re-running those moments, and keep supported edits provisionally until global validation decides.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Five steps: build evidence cards, write linked edits, replay each edit, keep supported edits across epochs, and commit through global validation.
- 6 · What they found ~3 min
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.