Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics

Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics. By Yuqing Yang and seven others, from the University of Southern California, Google, and one other institution. Published on the nineteenth of October. No code repository is linked to it.

A coding agent can answer an analytical question over a data lake. It finds the relevant data, cleans it, and joins it together. Then it throws all of that away, and for the next question it starts again from nothing. The discovery, the cleaning and the joining are redone each time, even when the next question needs much of the same work.

Tool evolution, which automatically builds reusable tools across tasks and refines them, is a natural fit for that kind of repetition. The obstacle is supervision. Getting a ground-truth label for a data-lake question means an expert has to solve it first, which is exactly the cost the agent was supposed to save. So the authors work in the unsupervised setting, with no labels at all.

Their system is TEAL, short for Tool Evolution for Analytics over data Lakes. It evolves a library of tools, together with skills for using them, for a given agent that does the actual answering, which the paper calls the Worker. What evolves is the library the Worker is handed.

Two other agents propose the tools, each drawing on a different kind of evidence. An Explorer works from the data lake itself. A Critic works from the Worker's trajectories, the record of what it actually did on earlier questions. The two sources complement each other: one sees what the data holds, the other sees where the Worker spent its effort.

Without labels, the hard part is deciding which proposals to keep. TEAL's rule has two conditions. On a rerun, the Worker must adopt the new tool without taking extra steps. And if the Worker's answer changes, the change must be supported by evidence in its trajectory. A proposal that fails either test is dropped. For cases where real questions are not available, the authors add TEAL-Q, which generates synthetic questions to drive the evolution.

Evaluation uses two benchmarks, KramaBench and LakeQA, with four Worker LLMs, each run under a step budget that caps how many steps the Worker may take per question. With tight budgets of ten or fifteen steps, TEAL improves every Worker on both benchmarks. Sonnet-5 at fifteen steps gains nine point three points on KramaBench and twenty-one point seven points on LakeQA.

A tighter budget also turns out to give better feedback for evolution. One example: a LakeQA library for Sonnet-5 is evolved at twenty steps, then deployed unchanged at thirty. It scores sixty-eight point three. The baseline scores fifty-nine point six, and a library evolved directly at thirty steps scores sixty-four point two. The library built under pressure beats the one built under the very conditions it is deployed in.

Finally, a library evolved on synthetic questions performs comparably to one evolved on real questions, which matters for the unsupervised case because synthetic questions are the cheap ones.

Two things make this work without a single label. Proposals come from two angles, the lake and the trajectory, and acceptance is judged by behaviour the system can observe: did the Worker take up the tool for free, and can its trajectory justify any new answer. The budget result adds a twist. Scarcity during evolution seems to sharpen the judgment of which tools deserve to stay. The authors hope the work prompts further study of what to evolve, and how to get feedback, when agents improve themselves without supervision.