PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving

PlannerForge. LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving. By Yuan Gao and seven others, from the Technical University of Munich and University College London. Published on the eighth of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.

Keeping autonomous driving safe is a critical challenge, and scenario-based testing is the systematic way Autonomous Driving Systems, ADSs, are validated. In practice, though, it is a fragmented modular pipeline. Scenario generation, retrieval, modification, execution of the ADS, and analysis of results are each handled by separate tools that barely interact. LLM agents have shown promise in parts of an ADS such as perception, planning and control, but no earlier work has covered the whole testing pipeline with one unified LLM-agent framework. The pieces exist. What has been missing is a single system that carries a scenario from the moment it is described, through running it against a planner, to making sense of what happened.

PlannerForge is that framework. It extends every stage of scenario-based testing, from Scenario Generation through to ADS Assessment, and adds two further stages enhanced by LLMs: ADS Enhancement and ADS Benchmarking. The testing pipeline becomes a single agent framework that can also improve and compare the planners it tests. Testing, in other words, stops being the last step. What it reveals feeds straight into making the planner better, and the improved planner can be measured again inside the same framework.

Evaluation covers ten off-the-shelf LLMs across every task, namely Generation, Selection, Modification, Module Routing, Planner Testing and Enhancement, under five prompt conditions. Best scores per task run from zero point eight eight up to one. Open-source backends between twenty and thirty-five billion parameters match commercial APIs on most tasks, and an open model such as Qwen3.6 at thirty-five billion matches them on three of the five tasks.

Each stage was also measured against an existing method for that stage, and the pipeline was run as a whole. Chained end to end, the modules retain eighty-three percent of seed queries with commercial models, and seventy-eight percent with open ones.

Against Scenario Factory 2.0 on generation from natural language, PlannerForge produces one hundred and ninety-three executable scenarios out of two hundred, against one hundred and forty-four. It realises between ninety-two and ninety-six percent of the requested city, road and vehicle attributes.

For selection, it beats BM25 at rank one, ninety-two percent against sixty-seven point five. For modification, it beats From-Words-to-Collisions on physically valid edits, at least ninety-four percent against thirty-one.

The last result is about improving planners. At a scale of four hundred, cost-tuning lifts planner success from fifty point four percent to seventy point two percent, and cuts collisions from nineteen percent to eight point four percent, with no fine-tuning specific to the driving domain.

What PlannerForge offers is continuity. Stages that used to live in separate tools now run inside one agent framework, and the chain does not stop at finding faults: it carries on into enhancing the planner and benchmarking it again. The results suggest that much of this can run on open models of moderate size rather than only on commercial ones, and that the planner gains came without any fine-tuning for the driving domain.