TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents
Krishanu Adhikary, Prashant Devadiga
National Payments Corporation of India (NPCI)
Published September 30, 2026 · on the feed October 29, 2026 · ▲ 11 · arXiv 2610.30161
The abstract
Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank's rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit. Adapting models to this domain requires measuring how well they follow the procedure, before and after training. Benchmarks usually judge only the records left at the end of a conversation. Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful. We present TauIndianBankBench, an action-oriented benchmark of 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. Besides the final records, it checks that the assistant took every required step and made no unrequested change. For every model, 6.5-12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change. Scoring both procedure and outcome shows more accurately how well language models perform in Indian retail banking.
The Brief
2:07A short podcast summary of the paper. Written and voiced by AI.
Transcript
TauIndianBankBench. Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents. By Krishanu Adhikary and one other, from the National Payments Corporation of India. Published on the thirtieth of September.
Many Indian banks serve their customers through FAQ pages and scripted chatbots. Assistants built on language models could answer faster and actually carry out requests, such as blocking a card. That only helps if they give correct information and follow the bank's rules. Those rules prescribe a procedure: a set of required steps, like obtaining a closure quote before closing a deposit. Adapting models to this domain means measuring how well they follow the procedure, both before and after training. Benchmarks usually judge only the records left at the end of a conversation. Yet many required steps leave the records unchanged. An assistant that skips them can still be judged a success.
TauIndianBankBench is built to see those skipped steps. It is action-oriented: besides checking the final records, it checks that the assistant took every required step, and that it made no change the customer did not ask for. Two kinds of failure become visible that an outcome-only check would miss. One is leaving out part of the procedure. The other is doing something unrequested along the way.
The benchmark contains one thousand tasks drawn from Indian retail banking. In each one, an assistant helps a simulated customer through a request. The scoring looks at procedure and outcome together, rather than at the end state alone.
The headline number is about how often the usual way of scoring gets it wrong. For every model tested, somewhere between six point five and twelve percent of sessions are judged successful when only the final records are checked, even though the assistant skipped a required step or made an unrequested change. That gap shows up for every model, not just a few. Scoring both procedure and outcome, the authors argue, gives a more accurate picture of how well language models perform in Indian retail banking.
A bank teller who hands over the right amount of money but never checks the customer's identity has still broken the rules, even if the ledger balances at the end of the day. A benchmark that reads only the ledger cannot tell the difference. This one watches the steps as well as the balance, and the share of sessions that change verdict when it does is the evidence that the steps matter.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A bank assistant can get the records right and still break the rules. Where would you look?
You want a language-model assistant to serve bank customers: block cards, close deposits, answer questions. The bank's rules prescribe required steps, like obtaining a closure quote before closing a deposit.
The road ahead
- 1 · The problem ~7 min If a banking assistant is judged only by the records left at the end, it can skip required steps and still look successful.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Score each session twice, on its final records and on whether every required call happened with no unrequested change; pass only if both hold.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build banking tasks from written policy, let an assistant serve a simulated customer, then score the session on records, required steps and unrequested changes.
- 6 · The experiment ~6 min You’ll guess: Does checking only the final records wrongly pass sessions that skipped a required step or made an unrequested change, and for which models?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.