SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale

SkillSeek. Revisiting Agent Skill Retrieval at Marketplace Scale. By Guanqun Yang and three others, from the Stevens Institute of Technology and Independent Researcher. Published on the thirtieth of September. Its code repository is public and real: about twenty source files.

Anthropic's Agent Skills package reusable procedural know-how for a language model agent into SKILL dot md directories. Open-source collections of them have grown past two hundred and thirty thousand skills, so the bottleneck is no longer writing skills but choosing among them. The standing answer has been to hand that choice to the agent: an LLM-mediated retrieval loop that rewrites queries and refines candidates inside the agent's decision loop, paying for language model tokens on every task. This paper's claim is that the standard information retrieval recipe does just as well, at essentially no extra cost.

The recipe is SkillSeek, an open-source two-stage skill retriever. A BGE-base bi-encoder feeds a small cross-encoder, and the whole thing is exposed over MCP. The comparison is a grid of four by eleven, crossing skill pool, backbone, and method, on the SkillsBench benchmark of eighty-nine tasks. Nothing in this retriever is new, and that is the point of the exercise. It is the standard two-stage arrangement from information retrieval, applied to skills, and it asks for no language model tokens to rewrite queries or refine candidates along the way.

Across that grid, SkillSeek reaches observed parity with the LLM-mediated loop of Liu and colleagues. Even plain bm25 on its own records a pass rate at or above their refined loop in three of the four settings. In the fourth, the small cross-encoder closes the remaining difference. The authors explain the pattern by a recall ceiling in the first stage. Cost moves the other way: total spend per trial falls from fifty-one point three zero US dollars to twenty-seven point five four, which lands within fifty cents of the baseline that uses no skills at all.

The authors bound the conclusion themselves. It holds under the SkillsBench tasks and the OpenHands harness they tested. Within that setting, the standard recipe is positioned as a strong default for retrieving agent skills, not as a replacement in every case. LLM-mediated retrieval remains a natural fit where deterministic methods fall short.

This is a paper that revisits a question rather than introducing a new kind of machinery, and its contribution is to put the standard recipe head to head with the agent-driven loop, on the same tasks and the same grid. At two hundred and thirty thousand skills, a reasonable instinct is that only an agent can choose well. The measured result points the other way, at least on this benchmark. A cheap retriever finds what the expensive loop finds, and using skills then costs barely more than not using them.