A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

A Closer Look at Agentic BBO. Benchmarking LLM Agents for Black-Box Optimization. By Ming Chen and ten others, from Nanjing University. Published on the eighth of October.

Black-box optimization shows up across science and engineering, wherever evaluating the objective is expensive and the number of evaluations is limited. Language model agents offer a new approach. They can combine the semantics of a task, computation, optimization tools, and decisions driven by feedback, and their integration with mathematically rigorous tools gives them real promise. The problem is comparison. Existing studies of agentic black-box optimization use different task domains and different system configurations. Their results are hard to compare, and the effect of any single design choice is hard to isolate. When two systems differ in their tasks, their tools and their budgets all at once, a better score from one of them could come from any of those differences. Without a common setting, the field cannot tell which ingredients of an agent are actually doing the work.

AgenticBBO-Bench is built to fix that. It is a cross-domain benchmark for agentic black-box optimization, and it measures every method under one unified evaluation protocol with a finite budget, so that agents, direct language model methods, and numerical optimizers can be scored on the same terms. A finite budget matches the setting that defines the problem in the first place, where every evaluation of the objective is costly and none can be wasted. Holding that budget fixed across methods is what makes their scores comparable.

The benchmark spans five domains: synthetic functions, hyperparameter optimization, database tuning, chip design, and molecular design. Beyond headline comparisons, the authors use it to study three factors that shape agent performance. The optimization tools the agent has. The task information and prior knowledge it is given. And the role the language model plays during search. They also add a five-task frontier challenge within the benchmark, and evaluate seven language models on it under the Codex agent harness.

Agentic optimization achieves higher family-averaged scores than direct language model methods in all five domains, and beats the best numerical optimizers in four of them. The factor studies are more mixed. Adding more numerical tools does not consistently help. Task semantics are broadly useful, while more specific priors are less reliable.

Numerical optimizers can effectively absorb the gains from search trajectories that the agent has already established, which says something about where the agent's value lies. On the frontier challenge, GPT-6 Astra and DeepSeek-V4.1-Flash sit on the Pareto frontier of performance against cost among the evaluated models.

A shared protocol turns scattered claims into a comparison. With all five domains on one budget, the picture that emerges is specific: agents help, but not by piling on tools, and not always through detailed priors. Much of what they contribute is a good starting trajectory, one that a classical optimizer can pick up and carry further. The code is public.