EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation
Harshavardhan Abichandani, Penny Chong, Jiyuan Shen, Gunraj Singh, Ashutosh Hathidara, Marcus Duigan Xing Yu, Jane Lo, Atin Ghosh and 2 others
SAP
Published September 21, 2026 · on the feed September 22, 2026 · ▲ 4 · arXiv 2609.24115
The abstract
Tool-calling LLM agents are increasingly deployed in enterprise applications. However, effective evaluation and optimization require high-quality, diverse task datasets that are often difficult to obtain due to privacy and other constraints. Existing synthetic task generation methods often produce generic tasks that ignore an agent's underlying state or database and fail to reflect real-world usage diversity. We propose EdgeGen, a synthetic task generation framework that extracts compliance rules from an agent's specification and uses them to generate database-grounded edge-case tasks designed to violate these rules. When combined with existing synthetic data generation techniques, EdgeGen enables agent improvement through finetuning and harness optimization. The resulting pipeline forms a fully automated closed-loop system that requires no human annotation. Finetuning on data generated by EdgeGen yields a consistent mean progress improvement of 2 percent to 42 percent on tau2bench airline domain, while other baseline methods show degradation for some models. On the other hand, for harness optimization, our method shows a mean progress improvement of 10 percent and 30 percent over the human-curated and base harnesses, respectively, for the Gemma-4-e4b model.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
EDGEGEN. Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation. By Harshavardhan Abichandani and nine others, from SAP. Published on the twenty-first of September. No code repository is linked to it.
Tool-calling agents built on language models are being deployed more and more in enterprise applications. Evaluating and improving them requires task datasets that are high in quality and diverse, and those are often hard to get, because of privacy and other constraints. Synthetic task generation is the natural substitute, but existing methods tend to produce generic tasks. They ignore the agent's underlying state or database, and they miss the diversity of how agents are really used. In the title's phrase, they stay on the happy path.
EdgeGen generates tasks aimed at the edges. It extracts compliance rules from an agent's specification, and uses those rules to generate edge-case tasks that are grounded in the agent's database and designed to violate the rules. The tasks are built to probe exactly the situations an agent's rules exist to handle.
EdgeGen is combined with existing synthetic data generation techniques, and the output feeds two routes to a better agent. One is finetuning the model. The other is optimizing the harness around it. The whole pipeline is a closed loop and fully automated, with no human annotation anywhere in it. Rules come out of the specification, tasks come out of the rules and the database, and improvements to the agent come out of the tasks, with each stage feeding the next.
For finetuning, the test is the airline domain of tau2bench. Training on data generated by EdgeGen gives a consistent improvement in mean progress, ranging from two percent to forty-two percent. Other baseline methods, in contrast, show degradation for some models, so the comparison is not only about the size of the gain but about whether there is one at all.
For harness optimization, the model is Gemma-4-e4b. There, the method improves mean progress by ten percent over the harness curated by humans, and by thirty percent over the base harness.
Taken together, EdgeGen turns an agent's own rulebook into its test set. The rules the agent is supposed to follow become the source of tasks designed to break them, grounded in the data the agent actually works with, and those tasks become the material for improving it, through the weights or through the harness. On the harness side, the automated result came out ahead of the one people had curated.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An agent that does fine on easy requests. The hard part is knowing when to say no.
You are improving an airline-support agent. It must refuse refunds that break eligibility rules and apply baggage policy by loyalty tier. Real customer logs are off limits for privacy, and writing tasks by hand is slow.
The road ahead
- 1 · The problem ~7 min Agents need practice tasks that test business rules, but real logs are private and machine-made tasks stay generic and easy.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min EdgeGen reads rules from the agent's policy, builds tasks meant to break them, ties each to database rows, and uses them to improve the agent.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Three steps: pull rules from the specification, generate database-grounded tasks that violate them, then improve the agent with those tasks.
- 6 · The experiment ~6 min You’ll guess: When models were finetuned on generated data for the tau2bench airline domain, how did EdgeGen data compare with other baseline methods?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.