AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning

AutoSFT. An Agentic System for Automated Data Research in Supervised Fine-Tuning. By Bingxiang He and twenty others, from Tsinghua University, the Beijing University of Posts and Telecommunications, and three other institutions. Published on the fifteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Language model agents are increasingly used to automate post-training, and when they are handed the full training stack they converge on supervised fine-tuning and spend most of their effort revising the training data. The claim here is about what it takes to automate that data work. Covering a diverse range of data actions matters more than having one stronger action. The contrast that motivates it is between existing systems and people. Existing systems give an agent one kind of control: selecting rows from a fixed pool, or calling a generator. A human data engineer adjusts proportions, prunes, repairs, replaces, and synthesizes. The authors design a data action space for all of those changes, then ask two questions. Can an automated system make the full range of changes rather than one of them? And what structure does that require?

The first test runs the loop with a single agent. It reaches only one action, adjusting category proportions. Meanwhile its context fills with execution output and is discarded partway through the search. That result motivates AutoSFT, a multi-agent system. A meta agent separates execution from analysis, gathers proposals from independent angles, and calls a synthesis subagent for capabilities that the existing pool cannot cover.

Components were added one at a time, and each one helped. The executor halves context loss. The analysis subagents lift the best IFEval score by six point five. Synthesis adds a further three point four on IFEval and two point nine on IFBench. The same loop transfers to code, where analysis adds three point seven on MBPP plus over a single agent that does not improve at all.

The comparisons are tightly controlled. Within each one, the model, the trainer, and the budget are held fixed. Every result comes from runs that retrain from the base model each round, so each component's effect can be attributed individually rather than inferred from a final number. The evidence comes from IFEval and IFBench, with one transfer to code measured on MBPP plus. What the findings suggest is drawn from those runs: a pattern across instruction benchmarks and one code benchmark, under a fixed model, trainer, and budget.

The order of discovery carries the argument. A single agent stalled on one action and lost its own context along the way. Separating execution, analysis, and synthesis opened up the rest of the action space, and the gains arrived with each addition. An automated data researcher, on this evidence, improves less by doing one thing better than by being able to do more of the things a human data engineer already does.