Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

Equal CER, Unequal Damage. Severity-Weighted Character Error Rate for OCR. By Nguyen Nguyen and one other, from the University of Rochester and Independent Researcher. Published on the fourteenth of October. Its code repository is public and real: about thirty source files.

Character Error Rate is the standard metric for optical character recognition and for vision-language models that read text. It has a blind spot built in. A harmless typo costs as much as getting a name wrong. Every edit is charged the same, whatever it does to the meaning. Two systems can therefore post an identical CER while doing very different damage to what a reader takes away, which is the point the title makes: equal CER, unequal damage.

The proposed metric, semCER, keeps what CER already does well: its alignment and its denominator stay the same. What changes is the charge. Each grapheme edit is given a severity, assigned by a language model acting as judge, and no word segmentation is needed. The authors are careful about division of labour. Normalized CER still scores a whole transcription, and at that level it tracks readers as well as semCER does, or better. semCER's role is to break that score down into named edits.

Validation is against human readers. The charges are compared with readers' ratings edit by edit, in English, Vietnamese and Chinese. A second test uses pairs selected where CER and semCER disagreed, and asks readers which side they take. The judge is then run across sixteen languages, to see how its sense of severity shifts from one writing system to another.

semCER's charges track reader ratings at the level of single edits, and the edits readers rate most severe land near the top of a sorted list. On the disagreement pairs, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across the languages, a wrong diacritic or tone mark costs almost nothing in Spanish but nearly as much as a wrong letter in Telugu. Where real words can be counted, a general vision-language model's wrong letters form one far above chance, while a classical pipeline's stay near chance.

Released with the paper are sixty-one million judged edits, the aggregated reader ratings, and a distilled scorer. The division of roles is clear. One number for the whole transcription is still a job for normalized CER. semCER answers the follow-up question of which errors inside that number actually hurt, and it does so in a way readers largely recognise.