EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

EvoSafeHarness. Evolving Model- and Domain-Specific Harnesses for Securing Agents. By Nanxi Li and six others, from Johns Hopkins University, the University of Wisconsin-Madison, and three other institutions. Published on the fifth of September. Its code repository is worked out: about ninety-five source files, with a readme.

LLM agents turn language into real-world effects. Once a model can act, safety becomes necessary on two fronts: indirect prompt injections, where hostile instructions arrive through the content an agent reads, and direct harmful requests from the user. Beyond the defenses built into the model itself, a system-level safety harness can add a separate enforcement layer. The trouble is how those harnesses are usually made. Experts design one once, and it is then applied across heterogeneous models and heterogeneous domains, as though one design fitted them all.

The authors argue that effective protection depends on the deployment. Models differ in how much enforcement they can take before their utility declines. Domains differ in the effects, the state, and the sequences of actions that need governing. So a harness strict enough for one model may over-block another, refusing work it should have allowed. And a policy that carries across domains may miss safety relations that are specific to one application. A single fixed design is too strict in one place and too loose in another.

EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for one frozen model in one target domain. The model itself is not retrained. Instead of an expert writing the rules once for everyone, the harness around the model is searched for, and the search is specific to that pairing of model and domain.

The search is joint. It covers two things at once: a natural-language policy, and executable code logic that enforces it. Three sources of information guide it. The first is the behaviour of the model. The second is the specification of the domain. The third is an adversarial review carried out in a fresh context, whose job is to reject rules that are specific to a benchmark rather than to the real safety problem. That review is what keeps the search from simply learning to pass the test in front of it.

The evaluation spans four families of agent benchmarks, and across them EvoSafeHarness reaches a stronger safety-utility frontier than fixed defenses designed by experts. On DecodingTrust-Agent, the average attack success rate drops from forty-five point six percent to ten percent, at a cost of three point three points of utility, and the method takes the best score in fourteen of fifteen cells.

On AgentDojo it reaches eighty-two point eight percent utility with an attack success rate of zero. That is twice the utility of CaMeL at the same operating point, and the same harness transfers unchanged to unseen AgentDyn suites. On Agent-SafetyBench it achieves the best score for every victim model. Under adaptive PAIR attacks, with a refinement budget of sixteen, the mean attack success rate stays below twenty percent.

The analysis splits the problem cleanly in two. The domain's semantics decide which safety relations, and which parts of the trajectory state, are needed at all. The model and its runtime behaviour decide how, and where, those relations should be enforced. A harness that respects both halves has to be fitted to its deployment, and fitting it automatically, rather than by hand, is the contribution.