Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Sho Kawano, Zehang Richard Li, Paul A. Parker
University of California, Santa Cruz
Published September 17, 2026 · on the feed September 22, 2026 · ▲ 4 · arXiv 2609.20758
The abstract
Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.
The Brief
2:42A short podcast summary of the paper. Written and voiced by AI.
Transcript
Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation. By Sho Kawano and two others, from the University of California, Santa Cruz. Published on the seventeenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
An AI system rarely performs the same everywhere. How well it does varies by domain: by task type within a benchmark, or by conversation type for an agent in deployment. So a useful evaluation is disaggregated, reporting performance per domain rather than one overall figure. Testing everything is expensive, which means evaluation rests on a labeled sample. The authors frame the evaluation set as a finite population, and the goal as accurate point estimates and interval estimates of each domain's mean.
The standard tools here are direct estimators, including prediction-powered inference, or PPI. A direct estimator uses only the labels from the domain it is estimating. Where a domain has few labels, that estimate is imprecise.
Statistics already has a field for exactly this situation, small area estimation, and the paper builds on it. The result is an integrated workflow with two halves: one for estimation, and one for validating which estimate to trust.
For estimation, the paper proposes prediction-powered smoothing, or PP-S. It is a Bayesian model fit to each domain's prediction-powered estimate. An extension, PP-TS, borrows strength across a reporting taxonomy, so domains that sit near each other in that taxonomy can inform one another.
For validation, the authors derive a new cross-validation score. It is design based and approximately unbiased, and its job is to choose among the direct estimators and the smoothed ones. Having more than one estimator on offer only helps if there is a principled way to decide between them, and this score is meant to be that way.
Two settings are studied, and in both every outcome was observed, so the estimates can be checked against the truth. One is a curated benchmark with verifiable grading. The other is deployed agent traffic graded by humans.
In both, the proposed estimators improve on the direct estimators, for point estimation and for interval estimation, with coverage near nominal. The validation score is compared with the obvious alternative, an independent validation sample. At the same sampling budget, the score selects an estimator as well as that independent sample does, and it estimates the selected estimator's error far more accurately.
In practical terms, this is about labels. The same labeling budget gets spent once rather than split between estimating and checking. Smoothing sharpens the thin domains, and the design-based score both picks the estimator and reports how wrong it is likely to be, without needing a separate held-out sample to do either.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Many domains, few labels each. That is the wall here.
You must report how well an AI system does in each of many domains: task types in a benchmark, conversation types for a deployed agent. One headline score would hide the differences.
The road ahead
- 1 · The problem ~6 min You need a score for every domain, but a labeled sample spread thin leaves small domains with too few labels for a precise estimate.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Fit a Bayesian model to every domain's judge-sharpened estimate so noisy domains borrow strength from the rest, then let cross-validation pick the estimator.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample and label units, compute each domain's prediction-powered estimate, smooth those estimates with a Bayesian model, then pick an estimator by cross-validation.
- 6 · The experiment ~6 min You’ll guess: When choosing among estimators, how did the new cross-validation score compare with spending the same budget on an independent validation sample?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.