Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills

Socratic Skill Learning. Interleaving Teaching and Practice for Transferable Agent Skills. By Letian Zhu and eleven others, from Nanjing University and Alibaba Group. Published on the thirteenth of October. Its code repository is worked out: about two thousand five hundred source files, with a readme and a licence.

Agent Skills improve language-model agents by making task knowledge reusable from one session to the next. Learning a Skill that transfers, though, needs evidence about a procedure and about the conditions under which it applies. A successful attempt at a task can leave that evidence untested. The agent may have succeeded without ever learning when the procedure would fail.

Socratic Skill Learning, or SSL, interleaves expert teaching with practice by a Student. An Educator guides the investigation, but the Student remains the only one operating the environment, and the only author of the Skill. The Student can question advice, test it, and adapt it, rather than simply receiving a finished procedure.

To test reuse beyond the task where a Skill was learned, the authors build SkillTransferBench. It holds thirty-five task families and four hundred and eight executable cases. Public variation specifications are paired with held-out evaluation in fresh solver sessions, so a Skill is judged by whether it works somewhere new.

On the main cohort of twenty-two families and one hundred and fifty-five cases, Fixed-Window SSL reaches a utility of zero point eight zero one. Without a Skill the figure is zero point seven two three. An adapted version of Trace2Skill, which distils trajectories, reaches zero point seven six one, and an adapted SKILL-KD, a contrastive distillation method, reaches zero point seven five one.

Across three model pairs on shared families, interleaved guidance outperforms both expert authoring and teaching after the fact. Process analysis links the guidance to what the Student writes and to how the package is revised, and a case study shows the Student testing and refining a proposed procedure for later reuse.

One cost grows over time. As the demand for new lessons falls, fixed-window review becomes more expensive relative to what it adds. Issue-Guided SSL responds by tracking unresolved questions across Skill revisions, and using them to select and focus review. On the main cohort it reaches a utility of zero point eight zero five, with sixty-one point seven percent fewer Educator tokens and sixteen point nine percent fewer total acquisition tokens than the fixed-window version.

The design choice at the centre is who holds the pen. The expert guides, but the learner runs the environment and writes the Skill, and that arrangement outperforms having the expert write it. Letting open questions decide when the expert speaks keeps the result while cutting most of the expert's tokens.