Human-Centred, Risk-Constrained Evaluation of Financial Document Recognition: Evidence, Critical Errors, and Selective Automation

Human-Centred, Risk-Constrained Evaluation of Financial Document Recognition. Evidence, Critical Errors, and Selective Automation. By Subodh Lonkar, working independently. Published on the second of November.

Financial document recognition is usually scored with character and word error rates. Those numbers say how much of the text came out right. They do not say whether the system preserved the financial facts that people and organisations actually act on. A reading can be nearly perfect character by character and still get the one amount, or the one sign, wrong.

The framework proposed here treats recognition as evidence recovery and decision support, and evaluates it in a way that is human-centred and constrained by risk. Its basic unit is the typed financial fact, linked to a visible region of the source document. It keeps three activities apart: verbatim transcription, deterministic normalization, and business interpretation. Around those sits a layered quality model, covering text, localization, layout, tables, fields, fact correctness, evidence grounding, uncertainty, robustness, and operational performance.

Acceptance is decided by non-compensatory gates. Each gate uses a one-sided uncertainty bound on critical-field errors, so a strong aggregate score cannot mask failures in amounts, identifiers, signs, periods, or the association of facts with entities. Selective automation, deciding which documents a machine handles and which go to a person, is formulated as a constrained routing problem. It accounts for residual risk, for the capacity of human reviewers, and for the cost of review. The framework also specifies procedures for annotation and adjudication, benchmark partitions designed to resist leakage, statistical estimands, subgroup analyses, and a reproducibility card.

The results are formal ones. They establish the limitations of compensatory scoring, where good performance in one place offsets bad performance in another. They establish that after zero observed errors, the upper bound on risk is still not zero. And they give a loss-sensitive decision rule for routing uncertain cases to human review.

The author is explicit about what is on offer. The contribution is theoretical and methodological, and no new model, dataset, or empirical performance result is claimed. What it supplies is a way of asking the question that matters for financial documents: not how many characters were right, but whether the facts someone will act on were recovered, with evidence, and with an honest bound on what might still be wrong.