Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
Hanlin Tian, Minhao Li, Yu Mi, Sihan Zhu, Zhao Yang, Yuxiang Wang, Hongquan Zhu, Qiufei Hu
DataGrids · Shanghai University · Peking University · Northwestern Polytechnical University · Nanyang Technological University · East China Normal University
Published August 31, 2026 · on the feed September 3, 2026 · ▲ 4 · arXiv 2608.30322
The abstract
Professional agent tasks often depend on conventions that are absent from public corpora, yet benchmarks rarely control whether an agent has access to those conventions. We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact containing private conventions, reference tables, and utility operators. Construction-time provenance, byte-identical task instructions across the provided- and withheld-artefact conditions, leak audits, and executable witnesses make dependence on the artefact explicit and testable. Across fifteen calibration tasks, one frontier agent configuration achieves a 68.0% pass rate with the artefact and 0% without it; on one task, a plausible but incorrect artefact also yields 0% across five trials. Deterministic solvers and rule corpora provide exact ground truth for structured tasks, while named criterion-level rubrics support outputs that cannot be checked by a single executable oracle. A configuration-relative calibration screen retains seven tasks satisfying our five-trial empirical knowledge-gating screen. These experiments validate the behavior of the construction protocol; they do not establish that the retained tasks improve post-training. We publicly release part of the task suite and supporting tooling at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction.
The Brief
2:08A short podcast summary of the paper. Written and voiced by AI.
Transcript
Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents. By Hanlin Tian and seven others, from DataGrids, Shanghai University, and four other institutions. Published on the thirty-first of August. Its code repository is public and real: fourteen source files.
Professional tasks given to agents often hinge on conventions that never appear in public text: private rules and ways of doing things that an agent cannot have read anywhere. Benchmarks rarely control whether an agent has access to those conventions. So when an agent fails, there is no way to tell whether it lacked the knowledge or lacked the skill. That is the question in the title.
The answer here is a knowledge-gated protocol for constructing tasks. It separates the task instruction from a compact artefact that holds the private conventions, reference tables and utility operators the task depends on. The same task can then be posed with the artefact provided, or with it withheld.
Several safeguards make that dependence explicit and testable. Provenance is recorded at construction time. The task instruction is byte-identical whether the artefact is given or withheld, so only the artefact differs. Leak audits and executable witnesses complete the set. For grading, structured tasks get exact ground truth from deterministic solvers and rule corpora. Outputs that no single executable oracle can check are graded instead against named rubrics at the level of individual criteria.
Fifteen calibration tasks were tried. One frontier agent configuration passed sixty-eight percent of the time with the artefact, and zero percent without it. On one task, a plausible but incorrect artefact also produced zero percent across five trials. A calibration screen, relative to that configuration and based on five trials, kept seven tasks that satisfy the empirical knowledge-gating test.
The authors are careful about what this shows. The experiments validate how the construction protocol behaves. They do not establish that the retained tasks improve post-training. What the protocol offers is cleaner failure analysis: a task where knowledge is gated by design, so that a miss can be attributed to ignorance or to incompetence. Part of the task suite and the supporting tooling is released.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An agent failed. Ignorance, or incompetence? Nobody can say yet.
You are building tasks for agents in professional domains. Many hinge on private in-house rules that never appear in public text.
The road ahead
- 1 · The problem ~7 min When an agent fails a task built on private in-house rules, you cannot tell whether it lacked the rules or lacked the skill.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Put the private rules in a separate artefact, run the identical instruction with and without it, and check answers exactly.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Split instruction from artefact, run both conditions, grade exactly, then keep only tasks that pass the knowledge-gating screen.
- 6 · The experiment ~6 min You’ll guess: With the private-conventions artefact withheld, how often did the frontier agent configuration pass the calibration tasks?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.