APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport

APort Vault. Benchmarking AI Agent Payment Authorization with the Open Agent Passport. By Uchi Uchibeke, from APort Technologies. Published on the eighteenth of September. Its code repository is worked out: about two hundred source files, with a readme and a licence.

Tool-using AI agents can move money, and APort Vault is a benchmark for payment authorization in such agents. It also carries a point of method, stated plainly by the author. The paper reports five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review.

The benchmark replays four thousand three hundred and seventy-one attacks that humans wrote against a live payment agent during a public capture-the-flag event. The replay spans fourteen models from eight labs, five policy configurations, and two replay tracks. Each is run with and without a deterministic pre-action check implementing the Open Agent Passport specification, OAP. In all, two hundred and twenty-five thousand nine hundred and sixty-four evaluations completed.

One constraint is built into the data. Each attack exists at exactly one configuration, so policy and attack cohort vary together. Payment requests are common, and their rate differs far more across configurations than across models. With the model alone, requests occur in ten point nine percent of evaluations at Level one, three percent at Level two, zero point one percent at Level three, and seventy-nine point four percent at Level four.

The one thousand two hundred and ninety-three Level four prompts were each evaluated on every model. Request rates run from seventy-one point two percent to eighty-four point three percent. Eight hundred and nine prompts, sixty-two point six percent, drew a request from all fourteen models, and every one of those ended in a successful payment to the level's allowlisted recipient.

The authorization boundary is where the conditions diverge. At Levels two to four, with the model alone, transfers to recipients the passport did not permit number one hundred and forty, out of seventy-six thousand eight hundred and forty-two. Behind the layer, they number zero, out of sixty-nine thousand two hundred and ninety-seven.

On matched triples of model, prompt, and track, sixty-eight thousand nine hundred and seventy of them, the count is one hundred and five against zero. That zero spans seven hundred and ninety source sessions, giving an upper bound per session of zero point three eight percent.

The zero was not bought by refusing payments. Behind the layer, twenty-five thousand three hundred and seventy payments executed. The policy denied one hundred and eighty-seven of the twenty-five thousand six hundred and forty transfer calls it evaluated, and one hundred and forty-eight of those denials were for a forbidden recipient.

The release includes all the evaluations, the level passports, the scoring code, and the analysis script. The benchmark's argument lives in how it counts. Models asked to pay at high rates, and payments went through, but transfers to recipients outside the passport dropped to zero once the deterministic check stood in front of the action. Keeping the events apart is what lets both of those facts be seen at once.