FDE-Gym: The Last Mile of AI Delivery

FDE-Gym. The Last Mile of AI Delivery. By Dongxu Yang and twenty-five others, from DeepLethe, Tsinghua University, and twenty-one other institutions. Published on the fifteenth of October. Its code repository is worked out: about one hundred source files, with a readme and a licence.

Most agent benchmarks hand the agent a well-specified task: an issue report, a specification, a task ticket. Real customer delivery does not arrive that way. Forward-deployed engineers, known as FDEs, have to clarify ambiguous requirements, work inside live systems, and uncover constraints that nobody stated. They are working with a customer, not from a ticket. That is the part of the job a well-specified task leaves out, and it is the part this benchmark is built to reach: the last mile, in the title's phrase, between a capable model and a delivered system that a customer actually benefits from.

FDE-Gym scores a delivery by how much it improves the customer's own success metric. The score is normalised so that zero means the system is unchanged and one means the reference solution. Asking is not free: penalties apply for stakeholder interactions, counted as messages, trial runs and waiting time. A task counts as resolved when it scores at least zero point eight with no incident. The design rewards two things at once. The delivery has to move the number the customer cares about, and the agent has to get there without leaning on the stakeholders more than it needs to.

The benchmark holds ninety-five end-to-end delivery tasks, drawn from twenty customer scenarios across eighteen sectors. Each was adapted by industry experts from a real-world delivery project. Agents work in simulated customer environments with scripted stakeholders and customer systems they are able to modify. They interact with the stakeholders and operate on production-like systems until they deliver a working solution. An online platform also lets humans practise the same skills under the same grading criteria.

Thirteen model configurations from seven providers were evaluated. In the realistic setting, where unstated constraints have to be discovered, the best configuration reaches a mean score of zero point five six. It resolves only fourteen of the ninety-five tasks. Even the strongest system tested, in other words, leaves the large majority of deliveries short of the resolution bar.

Discovering those constraints is the primary bottleneck. Performance drops sharply when models must uncover them without explicit guidance. Discovery alone, however, does not account for the gap. Consider the twenty-three tasks on which the best configuration obtained every key fact from the stakeholders. Even there, it reaches a normalised business uplift of zero point eight or above on only ten.

Two separate shortfalls come into view, and the benchmark is built so that they can be told apart. One is finding out what the customer actually needs. The other is turning complete knowledge of those needs into a delivery that moves the customer's own number. Together they mark, in the authors' words, a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.