IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga
National Payments Corporation of India
Published September 24, 2026 · on the feed September 28, 2026 · ▲ 20 · arXiv 2609.29167
The abstract
Banking assistants must use account-specific information to answer requests and, in many cases, take actions through tools. Evaluating only the final response misses important errors. An assistant may ask for information it already has, rely on stale context, select the wrong account, or write an invalid value after stating the correct one. We introduce IndicBankBench, a 799-case benchmark for Indian retail banking spanning five operational domains, a capability/refusal domain, and twenty primary axes. Cases are evaluated at four stages: safety, action and tool use, response adequacy, and advisory quality. Tool use and most safety checks are deterministic. A narrow resolver handles only ambiguous confirmation-before-write cases, while a separate LLM judge evaluates semantic response adequacy. We run every case three times and report strict pass^3, which requires success on all trials. Across the eleven evaluated models, strict reliability ranges from 43.7% to 58.2%, whereas at-least-once success ranges from 60% to 74%. This gap shows that at-least-once success can overstate dependable banking behavior. The case-level diagnostics also distinguish systems that ask unnecessary questions from those that act but fail to reconcile customer context or fully resolve the request. We release the cases, mock environment, and evaluation harness.
The Brief
2:22A short podcast summary of the paper. Written and voiced by AI.
Transcript
IndicBankBench. Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking. By Suvradip Paul and six others, from the National Payments Corporation of India. Published on the twenty-fourth of September. Its code repository is public and real: about twenty source files.
A banking assistant has to use information specific to the customer's account to answer requests, and often it has to act, through tools. Judging it only by its final response misses the errors that matter. An assistant may ask for information it already has. It may rely on stale context. It may pick the wrong account. Or it may state the correct value and then write an invalid one. None of those is guaranteed to show in the last message.
IndicBankBench measures an assistant at four stages rather than one: safety, action and tool use, the adequacy of the response, and the quality of any advice. It also measures reliability rather than luck. Every case is run three times, and the headline metric is strict pass-cubed, which only counts a case as passed if all three trials succeed.
The benchmark holds seven hundred and ninety-nine cases set in Indian retail banking. They span five operational domains, plus one domain for capability and refusal, and twenty primary axes. Grading leans on determinism. Tool use and most of the safety checks are checked deterministically. A narrow resolver is used only for ambiguous cases about confirming before a write. A separate language-model judge handles the one thing that needs semantic reading: whether the response is adequate.
Eleven models were evaluated. Their strict reliability ranges from forty-three point seven percent to fifty-eight point two percent. Their at-least-once success, counting a case if any trial succeeds, ranges from sixty to seventy-four percent. That gap is the paper's main point: at-least-once success can overstate how dependable an assistant's banking behavior really is.
The case-level diagnostics add a second kind of result. They separate systems whose failure is asking unnecessary questions from systems that do act, but then fail to reconcile the customer's context or fail to fully resolve the request. Those are different weaknesses, and a single score would blur them.
The authors release the cases, the mock environment, and the evaluation harness. The measure is built for a setting where one correct answer out of three attempts is not good enough, and on that measure, no evaluated model clears sixty percent.
Paper Quest · 7 chapters · about 35 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A bank assistant that is right most of the time is still wrong with someone's money.
You are judging an AI assistant for a retail bank. It checks balances, moves money and changes card settings through tools, using the customer's own account records.
The road ahead
- 1 · The problem ~6 min A bank assistant's last reply can look fine while the real mistakes happened earlier, inside its tool calls, or only by luck.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Grade each banking conversation in separate stages, keep checks mechanical where possible, and pass a case only if it succeeds on all three runs.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: Across eleven banking assistant models, how did strict reliability, passing a case on every one of its three runs, compare with succeeding at least once?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.