VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D, Srihari Bandarupalli, Santosh Kesiraju, Anil Vuppala
International Institute of Information Technology Hyderabad · Brno University of Technology
Published September 17, 2026 · on the feed September 18, 2026 · ▲ 31 · arXiv 2609.19879
The abstract
Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.
The Brief
2:03A short podcast summary of the paper. Written and voiced by AI.
Transcript
VākQA. A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering. By Bhavana Akkiraju and five others, from the International Institute of Information Technology, Hyderabad and the Brno University of Technology. Published on the seventeenth of September. No code repository is linked to it.
Question answering has advanced quickly with large language models, but mostly for high-resource languages, in text and in speech. For Telugu, spoken question answering had no benchmark, and nobody had measured how reliable automatic evaluation is in that setting.
Before benchmarking, the authors validated the evaluation methods against human judgements. Gemini as a judge comes closest to human ratings, but is strict in an uneven way. Open-weight judges systematically penalize correct Telugu answers whose surface form differs from the reference. A judge that marks a right answer wrong on wording alone would distort every result built on it.
VākQA is a Telugu spoken question answering benchmark of two thousand and one factoid question-answer pairs across six domains. It comes with two point five three hours of speech audio, bilingual transcriptions, and reference answers verified by humans. Using the validated setup, proprietary and open-weight models are compared across input modality, language and domain. Because each question exists as speech and as text, in Telugu and alongside a translation, the effect of each change of modality or language can be looked at on its own.
Three observations stand out. Telugu phrasing keeps a cultural specificity that is lost in translation. Speech input introduces phonetic confusions that change what a question means. And in cascaded systems, errors from speech recognition and from machine translation compound as they pass along. Each of the three points to a different place where meaning can slip: in translation, in hearing, and in the handoff between systems.
VākQA is publicly released. Its first result is about measurement itself: in a language like Telugu, the choice of judge can decide whether a correct answer counts.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A new language, a new benchmark, and a quiet question: can you trust the grader?
You want to measure how well models answer factoid questions asked aloud in Telugu. There is no spoken Telugu benchmark, so you will build one.
The road ahead
- 1 · The problem ~6 min To test models on spoken Telugu questions you need a benchmark and a grader you can trust, and neither exists yet.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Test the graders against native Telugu speakers first, then benchmark models on the same questions in several forms.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Mine spoken quiz questions, verify and translate them, test the graders against people, then benchmark models one change at a time.
- 6 · The experiment ~6 min You’ll guess: When automatic graders were checked against human ratings of Telugu answers, which kind came closest to the humans?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.