ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost

ParetoSaddler. Pareto-Guided Optimization of Agent Harnesses for Quality and Cost. By Xingzhi Qian and fifteen others, from University College London, the Pohang University of Science and Technology, and four other institutions. Published on the first of November.

Optimizing the harness around a language model agent can improve it substantially without touching the model's weights. Existing methods, though, mostly optimize response quality alone. Stretching them to optimize quality and cost together raises a problem. If the deployment preference is applied while the search is still in progress, it can reject candidates too early, including ones that would have been valuable stepping stones for later improvements. A candidate that is not yet the one to deploy may still be the one to build on.

ParetoSaddler resolves that mismatch by keeping two decisions apart. Promoting an intermediate candidate during the search is one decision. Choosing the harness to deploy at the end is another. The deployment preference governs only the second.

At each iteration, an optimizer updates a parent harness to produce a candidate. The two are compared over repeated executions, and Bayesian bootstrap Pareto support is used to promote candidates that show credible progress in either quality or cost. A candidate need not improve both. Repeating the executions matters because a single run could make a weak candidate look strong by chance; the comparison asks for progress that is credible. Promoted candidates then get a broader development evaluation and stay available as parents for later optimization steps.

The evaluation uses ClawMark and GAIA2, with splits that are disjoint by domain. On ClawMark, quality rises from seventy point eight six percent to eighty-one point three five percent, while cost per task falls from one point one seven to one point zero one. On GAIA2, quality rises from fifty-six point two two percent to sixty-two point eight nine percent, and cost per task falls from zero point seven eight to zero point four five.

The Pareto-selected harnesses also reach higher quality at lower cost than harness optimization aimed at quality alone. Further analysis shows two effects at work. Pareto-aware optimization keeps intermediate candidates that the deployment preference would have thrown away but that can support later improvements, and the Bayesian bootstrap screening filters out candidates whose gains are only weakly supported across runs.

The design rests on a distinction between what a search should keep and what a deployment should ship. A candidate that is merely cheaper, or merely better, can still earn its place in the lineage, and on both benchmarks that patience ends in a harness that improves on both axes at once.

ParetoSaddler. Pareto-Guided Optimization of Agent Harnesses for Quality and Cost. By Xingzhi Qian and fifteen others, from University College London, the Pohang University of Science and Technology, and four other institutions. Published on the first of November. This is the long version: the whole paper rather than the abstract.

An agent's harness is everything wrapped around the model: the prompts, the tools, the middleware and the control logic that sit between it and its environment. Automatic harness optimization has an outside optimizer read execution traces and patch the harness, with the weights untouched. Most such methods chase quality alone, but deployment also pays for tokens. The obvious extension is to apply the deployment preference at every step, and keep an update only if it already sits on the trade-off you mean to ship. The paper argues that this is the mistake, because for long-horizon agents cost is tangled up with progress. A harness can look cheap because some of its runs quit early. The patch that fixes the early quitting raises quality and raises cost too, through more reasoning and more environment interaction. Reject it, and no later patch can build on it and trim the inefficiency it exposed.

The first piece of background is staged evaluation. At each iteration, an optimizer takes a parent harness, diagnoses traces from a small mini-batch of training tasks, and proposes a candidate. The candidate is first screened cheaply against its parent on that mini-batch. Only candidates that pass, a step the paper calls promotion, get the broader development evaluation and become available as future parents. So the screening rule decides what the search can build on.

The second is noise. Across twenty ClawMark tasks with six runs each, the mean within-task coefficient of variation was eighteen point eight percent for quality and twenty-five point six percent for cost. An apparent improvement may be luck.

ParetoSaddler sits on top of an existing optimizer, AutoSaddler. The parent and the candidate are each executed three times per screening task. Rather than comparing averages, it uses the Bayesian bootstrap. In each draw, every task receives random Dirichlet weights over its three runs, with the same weight applied to a run's quality and to its cost, and the tasks are then averaged equally. Each draw then computes the quality difference in absolute points, and the cost change relative to the parent's cost.

A draw counts in favour of promotion if either of two things holds. Quality went up while cost stayed within an allowance. Or cost went down while quality fell by no more than an allowance. Progress along either axis is enough, whatever the final deployment wants. In the experiments the quality allowance is ten points and the cost allowance is set to infinity, because a large local cost increase can reflect genuine progress. Ten thousand draws are taken, and the candidate is promoted only if at least seventy percent of them support it. The draws reuse the recorded runs, so they need no extra model calls.

Promoted candidates are evaluated on the development set and added to a pool of every development-evaluated harness, from which a Pareto archive is computed. The pool keeps candidates outside the archive too. The deployment preference may steer the next parent, but is never a hard filter. Only when the budget runs out is it applied, once, to choose the harness or the set of harnesses to ship.

Test sets come from domains never seen during optimization. On ClawMark, the quality-oriented Pareto harness raises quality from seventy point eight six to eighty-one point three five percent, while cost per task falls from one dollar seventeen to one dollar and one cent. On GAIA2, quality rises from fifty-six point two two to sixty-two point eight nine percent, and cost falls by forty-two point three percent. It also beats the quality-only AutoSaddler baseline by seven point four nine points on ClawMark and five point five six on GAIA2, while costing less. Adding cost led the search to better quality.

The comparison that most directly tests the thesis is a controlled gate swap in the cost-minimising setting, replacing the Pareto gate with one that promotes only supported cost reductions. The Pareto gate promotes eleven candidates. The cost gate promotes three. Then iteration eleven lifts mini-batch quality to eighty-seven point six two percent while raising cost by ten point five percent. The cost gate would have rejected it. Kept, it led to a better test operating point. A second swap tests the statistics: replacing the bootstrap with a plain comparison of means drops the quality-oriented harness to seventy-six point five three percent quality, at a higher cost.

The paper has no limitations section. Its scope statement says it does not evaluate fairness, harmlessness or safety, so gains in quality and cost should not be read as readiness for real deployment. The appendix adds that with three runs per task, rare outcomes can go unobserved. The single-run versus three-run ablation is not compute-matched, since three runs simply spend more. Each method's main result comes from one optimization run; a second run was done only for ParetoSaddler, and only on GAIA2, where its gains held. The optimizer itself costs more: on GAIA2, nine dollars sixty-two per patch against six dollars and one cent for AutoSaddler.

The idea to carry away is that what you explore and what you deploy are different questions, and a search that answers them with the same rule throws away its own stepping stones. ParetoSaddler keeps updates with credible progress on either axis, judges credibility by resampling repeated runs, and saves the deployment preference for the end. On both benchmarks, that patience bought harnesses that were better and cheaper at once.