AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation

AGO AI Quality Gate. Evidence-First Release Decisions for Retrieval-Augmented Generation. By Giulio Zeloni and five others, from Protom Group. Published on the first of October. No code repository is linked to it.

An enterprise running retrieval-augmented generation keeps meeting the same operational question about each new system version. Promote it, revise it, or block it. That call has to be made on evidence that is incomplete, and the metrics feeding it come from LLM judges, which are themselves fallible. A release decision built on a judge nobody has checked, and on data with holes in it, can look confident while resting on very little.

AGO, the AGO AI Quality Gate, is an evidence-first quality-gate framework, and the authors report on it as something deployed in industrial RAG assessment engagements. Its starting stance is that missing data and judge errors are not noise to be smoothed over. They are outcomes in their own right, and the decision model has room for them.

Four components make up the framework. The first is a four-state decision model that treats missing data and judge errors as explicit outcomes. The second is layered scoring, which combines deterministic checks, local guardrails, and structured LLM evaluation. The third is a stratified beta-binomial gate that quantifies regression risk probabilistically, rather than reading a single score. The fourth is a mandatory meta-evaluation protocol: the LLM judge has to be validated before it is allowed to influence any decision.

Engagement data is proprietary, so the judge layer is evaluated on RAGBench, a public benchmark of one hundred thousand annotated RAG traces drawn from twelve datasets. Each judge sees identical stratified test samples, twelve hundred per judge. A low-cost judge, gpt-4.1-nano, detects non-adherent answers barely above chance, with an AUROC of zero point six zero three. It does this while producing flawless protocol output. The output looks right; the judgment underneath it is close to a coin toss.

gpt-4o does much better, reaching an AUROC of zero point seven eight three. Yet its performance per domain still ranges from zero point six two at the low end to zero point eight eight at the high end. A separate fixed-seed gate study covers three scenarios: regression, no change, and improvement. It quantifies unsafe promotion, false-alarm cost, and improvement throughput. Under regression, the decision-grade profile brings unsafe promotion down to between twenty-two point two percent and thirty-five point one percent. A naive gate sits between twenty-nine point three percent and forty-one point eight percent.

The authors read these results as support for two design choices. Judge quality has to be measured per engagement, because a judge that does well overall can still vary widely from one domain to the next. And a point estimate by itself is not a release decision. A judge whose output is flawless in form can still be barely above chance in substance, which is why the protocol validates the judge before it is allowed to count.