Calibrated Typed Decisions for Entity, Relation, and Event Extraction

Calibrated Typed Decisions for Entity, Relation, and Event Extraction. By Yang Wang and four others, from the University of Manchester, the Mayo Clinic, and one other institution. Published on the twelfth of October. Its code repository is worked out: about ninety source files, with a readme and a licence.

Information extraction models often do well on the datasets they were trained on, and then struggle when moved to a new domain or a new set of annotation guidelines. Letting the user specify the types to extract is a natural first response, but the authors argue that supporting user-specified types alone does not resolve the problem.

CalOn is their attempt at one model for all of it: a unified encoder of zero point six billion parameters that handles entity, relation, and event extraction across datasets and schemas. The central design choice is about how a decision gets made. Rather than judging each requested type in isolation, the model considers the requested types together and uses their combined evidence to pick out relevant candidates first, and only then assigns a type.

Text and type definitions are encoded jointly. From that encoding the model directly identifies entity spans, relations between spans, and event triggers with their arguments. Because all three tasks run through the same decision, they share a common probabilistic formulation, and every extracted mention stays anchored to the input text. Training draws on diverse sources, and supervised learning is supplemented with a reward that encourages correct extractions and penalises incorrect ones. That training introduces a probability bias, and the authors add a correction for it.

Across forty benchmarks, CalOn outperforms the evaluated baselines on most of them. It leads every reported relation extraction benchmark, and it achieves the highest average F1 for event triggers, and for event arguments when gold triggers are given. Because relations are handled as pairs of spans, the same model also performs strongly on two tasks outside its core three: aspect sentiment triplet extraction and pronoun resolution. The bias correction improves confidence estimates for entities, relations, and event arguments after calibration, with little change in F1.

The word calibrated in the title carries real weight. Beyond extracting the right spans, the model's probabilities are meant to be trustworthy, and the correction delivers better confidence without paying for it in accuracy. A single span-anchored formulation is what lets one small encoder stretch across this many tasks.