TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

TeleAntiFraud 2.0. A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection. By Huiyuan Liu and thirteen others, from the People's Public Security University of China, J D Technology, and four other institutions. Published on the seventeenth of September. No code repository is linked to it.

Telecom fraud scripts change quickly, and they are often written to sound like routine service calls. That places two requirements on audio-based fraud evaluation. A benchmark has to take in newly observed scam patterns without overwriting test sets already established. And it has to tell fraud apart from lawful calls in the same domain, rather than leaning on negative examples that are simply about different topics. A benchmark whose non-fraud calls are about something else entirely lets a model succeed by noticing the topic, which is not the skill a fraud detector needs when the scam is dressed as a service call.

TeleAntiFraud 2.0 is built to meet both requirements. It is refreshable, so new scam patterns can enter. It is grounded in profiles, so scenarios are anchored to particular situations. And it is based on audio, so models are tested on speech rather than only on transcripts. Evaluation runs under a monthly frozen protocol: each month produces a new set, and earlier sets are not overwritten. Each frozen set holds nine hundred Chinese calls, made up of six hundred fraud cases and three hundred near-domain non-fraud cases.

Construction runs through the Mixed-Tree Anti-Fraud Generation Pipeline. It takes abstracts of fraud cases found online and turns them into scenarios grounded in profiles. It expands those through mixed-tree generation. It realizes fraud and non-fraud dialogue paths under shared contexts. Validated dialogues are then rendered as speech with matched roles. Finally, for each monthly set, the audio, labels, prompts, manifests, and provenance records are frozen, so that earlier sets stay as they were when new patterns arrive.

Controlled text experiments show why the negatives matter. Three classifiers score a perfect macro-averaged F1 when tested against unrelated or ordinary negatives. Against near-domain sibling negatives, they drop to between zero point six five and zero point six eight. Full-set evaluations, on audio directly and with speech recognition feeding a large language model, reveal more problems: shortcuts that exploit class priors, prediction collapse, and sensitivity to which snapshot is tested. A perfect score, in other words, can be a property of the negatives rather than of the classifier, and a single aggregate number can hide a model that has collapsed toward one answer.

From this the authors draw two requirements for anyone evaluating audio telecom-fraud models under realistic, confusable conditions. Build the negatives from the near domain, and report with collapse in mind. The released artifact includes the construction code, evaluation scripts, manifests, and documentation, so each month's set can be rebuilt and checked.