Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations
Yonghong Zhang, Yong Xie, Isabel M. Parra, Ricardo Correia
Universidad Autónoma de Madrid · Spanish National Research Council (CSIC)
Published September 25, 2026 · on the feed September 28, 2026 · ▲ 5 · arXiv 2609.30867
The abstract
Difference-in-differences (DID) studies are widely used to evaluate climate policy, but assessing the evidence supporting their identification assumptions remains challenging. We introduce ARGUS, a structured language-model pipeline that audits reported evidence against an eleven-dimension assumption-implication-evidence rubric and abstains when relevant evidence cannot be retrieved. We evaluate ARGUS using injected flaws, economics papers, and a small pilot with reconciled labels. On the 11-flaw benchmark, ARGUS detects 73% of planted flaws, compared with 18% for a keyword-based pipeline. Across 26 economics papers, ARGUS abstains on about 40% of paper-dimension assessments for lack of retrievable evidence. In a five-paper pilot with labels reconciled by two annotators, it assigns a higher risk level than the labels on 25 of the 33 assessments it completes. A rule fixed before the labels arrived removes most of this in-sample; weighted agreement stays low. ARGUS provides evidence-linked risk reports that localize potential weaknesses for expert review, without adjudicating causal claims. Code and data: https://github.com/yonghongzhang-io/ARGUS
The Brief
2:06A short podcast summary of the paper. Written and voiced by AI.
Transcript
Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations. By Yonghong Zhang and three others, from the Autonomous University of Madrid and the Spanish National Research Council. Published on the twenty-fifth of September. Its code repository is worked out: about eighty source files, with a readme and a licence.
Difference-in-differences studies, known as DID, are widely used to evaluate climate policy. Their conclusions rest on identification assumptions, and assessing the evidence that supports those assumptions remains challenging.
ARGUS is a structured language-model pipeline for that assessment. It audits the evidence a study reports against a rubric of eleven dimensions, each one tying an assumption to its implications and to the evidence that bears on it. When relevant evidence cannot be retrieved, ARGUS abstains.
What it produces is an evidence-linked risk report. The report localizes potential weaknesses so that an expert can review them. It does not adjudicate the causal claims themselves: that judgment stays with the reader. ARGUS was evaluated three ways. With injected flaws, on real economics papers, and in a small pilot with reconciled labels.
On the benchmark of eleven planted flaws, ARGUS detects seventy-three percent of them. A keyword-based pipeline detects eighteen percent. Across twenty-six economics papers, ARGUS abstains on about forty percent of paper-dimension assessments, for lack of retrievable evidence.
The pilot used five papers, with labels reconciled by two annotators. Of the thirty-three assessments ARGUS completed, it assigned a higher risk level than the labels on twenty-five. A rule fixed before the labels arrived removes most of that gap in-sample. Even so, weighted agreement with the labels stays low.
Read together, the numbers describe a tool with a particular temperament. It finds far more planted flaws than a keyword approach, it declines often when evidence is missing, and in the pilot it rated risk higher than the human labels did. Its stated role is deliberately narrow: pointing an expert at where a study might be weak, not deciding whether the study is right. Code and data are public.
Paper Quest · 7 chapters · about 42 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A causal claim is only as strong as its assumptions. Checking them is slow.
You review climate-policy studies that compare treated and untreated regions before and after a policy. Each conclusion rests on assumptions, like the two groups trending alike before the policy.
The road ahead
- 1 · The problem ~7 min A climate-policy study's verdict is only as good as the evidence it reports for its assumptions, and checking that evidence is slow and easy to fake.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Audit a study assumption by assumption against its own retrieved passages, in fixed coded steps, abstaining when nothing relevant is found, and report risks for experts.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min The audit runs fixed stages: decompose into eleven dimensions, retrieve evidence, abstain if none, judge what remains, and report risks for an expert.
- 6 · The experiment ~7 min You’ll guess: On a benchmark of eleven planted identification flaws, what share did the auditor detect, where a keyword-based pipeline caught few?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.