SkillSpec: Intent-Masked Specification Reasoning for Agent Skill Correctness

SkillSpec. Intent-Masked Specification Reasoning for Agent Skill Correctness. By Yizhuo Zhang and five others, from Beihang University. Published on the fifth of September. Its code repository is worked out: about seventy source files, with a readme and a licence.

Autonomous agent systems lean more and more on reusable skills. A skill is an abstraction that consolidates experiential knowledge and domain expertise, and in practice it bundles free-form instructions together with heterogeneous resources. Making sure such a thing is correct turns out to be hard, and not for the usual reasons. The failures go beyond ordinary code defects. They include subtle semantic inconsistencies, such as intent conflicts, where one part of a skill wants something another part does not. Those show up as silent failures, because the underlying model papers over them. And correctness here is not an absolute property. A skill is correct relative to the task boundaries it was meant for, and relative to how far it is supposed to generalize. A checker has to be grounded in that intent, or it has nothing to check against.

SkillSpec treats skill correctness as a specification reasoning problem, in the Hoare style. The idea is to set what a piece of a skill is expected to do next to what it actually does, and to look for the places where the two disagree. The expectation comes from declared intent. The observed behavior comes from what is encoded. A defect is a gap between them.

The first step is structural. SkillSpec transforms a heterogeneous skill repository into one unified graph, in which descriptions, instructions and code artifacts are aligned with one another. Then, for each node in that graph, it derives two kinds of specification. An ExpectSpec is derived from the declared intent surrounding the node. FactSpecs are inferred from the behavior the node encodes, and they are inferred while the intent is only partially disclosed.

That partial disclosure is controlled by an intent mask. The mask regulates which views the reasoning can see: a holistic view, a lineage view, a neighborhood view, and a local view. The reason for rationing context is a balance between two failure modes. Too much context introduces bias, while too little leads to inference the evidence cannot support. SkillSpec reasons jointly over these views to flag candidate defects, and then validates each candidate automatically in an isolated sandbox.

The evaluation covers five hundred and fifteen real-world skills, drawn from SkillsBench and from widely downloaded repositories. On those, SkillSpec identified seven hundred and sixty-three defects that were manually confirmed, spread across two hundred and thirty-nine skills, at a precision of sixty-one point two percent.

A node-level analysis across multiple model families splits the picture by node type. For code nodes, specification reasoning is consistently reliable. Plain-text nodes remain a major bottleneck. And when the defects are located, most of them arise at the boundaries between declared intent and implementation.

Where the defects cluster brings the argument full circle: exactly where what a skill says it is for meets what it actually does, which is the seam an explicit specification is built to expose. The authors read this as evidence that explicit specifications offer a practical foundation for quality assurance of skills in real-world agent ecosystems, with prose instructions still the hardest part to reason about.