Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions
Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr
Vanderbilt University · University of Southern California · Adobe Systems
Published September 29, 2026 · on the feed October 2, 2026 · ▲ 9 · arXiv 2609.38593
The abstract
Skills are external artifacts that Large Language Models (LLMs) consume at inference time to improve their performance on specialized domains by incorporating relevant procedural and domain knowledge. Expert-authored skills are expensive to produce, and the resulting artifacts are not optimized for the specific model that consumes them, whose failure modes can vary with version, scale and training. In addition, emerging tasks may fall outside the scope of existing skill libraries, creating a need to develop new skills before curated training data become available. Recent works have explored automated skill optimization through reflection, but they require a curated, in-distribution training set, which users might not always have. To address these limitations, we present Prompt2Skill, a framework that builds skills from natural-language task description alone. From the prompt, the system derives a task specification, discovers or synthesizes datasets, and refines the skill in a closed loop of reflective editing. Across four domains spanning question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill consistently outperforms the direct prompting baseline, achieving an average improvement of 10.8 across open-source and frontier models.
The Brief
2:27A short podcast summary of the paper. Written and voiced by AI.
Transcript
Prompt2Skill. Unsupervised Skill Optimization From Natural Language Instructions. By Bo Ni and four others, from Vanderbilt University, the University of Southern California, and one other institution. Published on the twenty-ninth of September. Its code repository is worked out: about eighty-five source files, with a readme and a licence.
Skills, in this sense, are external artifacts a large language model consumes at inference time. They carry procedural and domain knowledge that lifts its performance in specialized domains. Getting good ones is hard in three ways. Experts are expensive to have write them, and what they write is not tuned to the particular model that will read it, even though a model's failure modes shift with version, scale and training. Emerging tasks can fall outside every existing skill library, so new skills are needed before any curated training data exists. Recent work automates skill optimization through reflection, but it depends on a curated, in-distribution training set, which a user may simply not have.
Prompt2Skill builds a skill starting from nothing but a natural-language description of the task. No curated dataset is handed to it, and no expert draft. The prompt is the only input. That choice is aimed squarely at the limitations just listed. It needs no expert author, and it does not wait for a curated, in-distribution training set, so it can start on an emerging task for which no curated data exists yet.
From that prompt the system works in stages. It first derives a task specification. It then discovers existing datasets or synthesizes new ones to match. Finally it refines the skill in a closed loop of reflective editing, so the data it found or built stands in for the training set that earlier reflection methods required. Specification, data, and reflective editing: three steps, all of them downstream of a single prompt.
Evaluation covers four domains: question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning. Across all of them, Prompt2Skill consistently outperforms the direct prompting baseline, and the average improvement is ten point eight, measured over both open-source and frontier models. The abstract gives that figure without naming its unit.
Where does the training data come from, if nobody supplies it? Prompt2Skill's answer is that the system finds or makes its own, starting from the same sentence a user would have typed anyway, which removes the prerequisite that kept reflective skill optimization out of reach for new tasks.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A new task, a model that could do it, and nothing to train on but a sentence.
You want a language model to handle a new kind of task: read a Wikipedia passage and answer questions about it. A skill would help, a markdown file of procedures the model reads at inference time.
The road ahead
- 1 · The problem ~6 min You need a good skill file for a new task, but you have no expert, no labelled data, and only one sentence describing the task.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Start from the task description alone, find or build practice data for it, and refine the skill against the target model's failures, keeping only edits that clearly win.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Three stages run in order: turn the prompt into a task specification, gather matching data, then refine the skill by reflective editing.
- 6 · The experiment ~6 min You’ll guess: Building skills from only a task description, how large was Prompt2Skill's average improvement over direct prompting across models?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.