AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

AMLworld-Compact. Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis. By Natapong Nitarach and five others, from SCB DataX at SCBX Group and Independent Researcher. Published on the ninth of October. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.

Evaluating large language models on AMLworld is expensive, even on its smallest test splits. There are over a million targets, and each one needs its own graph prompt. Sampling a subset would cut the cost, but a small balanced subset changes the unweighted precision and F1 scores, so the numbers no longer describe the full test. A cheap evaluation that reports the wrong precision is not much of a saving. The task, then, is to shrink the test while keeping the scores it yields tied to what the full test would have said.

AMLworld-Compact is an evaluation set built to keep the cost down without losing the full-test picture. It retains every illicit test transaction and samples benign transactions from strata defined by a supervised score. What it estimates are full-test confusion counts. Recall comes out exact, while precision and F1 are ratio estimates.

The protocol, called HT-Coreset, reduces evaluation targets by two hundred and seventy-one times on HI-Small, and by six hundred and eleven times on LI-Small. Horvitz-Thompson weights make the confusion-count estimates unbiased for any fixed set of predictions. To test this, a held-out study uses Graph Feature Preprocessor features: one boosted-tree family defines the strata, and the other is evaluated, over five hundred draws per configuration.

At the released sizes, F1 error stays between zero and two point five eight percentage points of root mean squared error. Uniform sampling with the same weights gives between four point four four and seven point four one. Conservative intervals remain broad, though. Under the evaluated graph prompts, seven open-weight language models have weighted precision near the baseline of predicting every transaction as illicit.

An outcome-stratified audit covers one thousand few-shot outputs on HI-Small, from one run per model. Lexical checks pass for eighty-four point five percent of correct binary verdicts, and for fifty-six point two percent of incorrect ones. Those markers, then, do not establish correctness. A surface check that passes for most wrong answers as well as most right ones cannot stand in for checking the verdict itself.

The evaluation sets, construction code and audit tools are released on Hugging Face and on GitHub. The result is a cheaper way to score models on this task that still says something about the full test, and an early reading that the models evaluated are not yet doing much better than flagging everything.