BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence
Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri
UIUC · Microsoft Research · Microsoft
Published September 16, 2026 · on the feed September 21, 2026 · ▲ 28 · arXiv 2609.20886
The abstract
Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.
The Brief
3:15A short podcast summary of the paper. Written and voiced by AI.
Transcript
BI-Agent and BI-Bench. Towards Automating End-to-End Business Intelligence. By Chuxuan Hu and five others, from the University of Illinois Urbana-Champaign, Microsoft Research, and one other institution. Published on the sixteenth of September. Its code repository is worked out: about eight hundred source files, with a readme.
Business intelligence is a cornerstone of how enterprises make decisions, and it is used widely in software such as Power BI and Tableau. A traditional workflow asks a lot of the user before any question gets answered. First they identify the relevant tables. Then they perform data transformations. Then they build join relationships. Only after all three can they answer the business question they came with. The question itself may be simple. Everything standing in front of it is not. Those steps can be complex and time-consuming, and they are what makes business intelligence challenging.
Large language models are strong at working with data, so the authors ask a direct question: can such a model answer business intelligence questions end to end, without users manually performing the tedious preparation? Until now, no benchmark had set out to study that systematically.
BI-Bench is presented as the first benchmark to do so. What it measures is the whole job rather than its last step. Most of the effort in a real project sits in the preparation, and a benchmark that skipped it would be measuring the easy part. A model is not handed tables that have already been chosen, cleaned, and joined. It has to get from the raw project to the answer, doing the preparation along the way.
The material is real rather than invented. The authors harvest a large collection of real-world business intelligence projects from public sources. From real user dashboards in those projects, they manually extract pairs of questions and ground-truth answers. The ground truth, then, comes from dashboards that people actually built on this data, and the questions are ones those dashboards were built to answer.
The headline finding is that even frontier models perform poorly on BI-Bench, with accuracy below fifty percent. To address those limitations, the authors design BI-Agent, an agent augmented with tools. It decomposes a business intelligence workflow into subtasks over structured data, such as search, join, and transform, and orchestrates specialized data-management methods across the stages of the workflow. The steps a user once did by hand become subtasks the agent works through.
They also build a post-training framework that synthesizes training trajectories from real business intelligence projects, so BI-Agent can be further trained with supervised fine-tuning and with reinforcement learning. The agent brings accuracy gains of up to forty percentage points with vanilla language models, and the post-trained BI-Agent yields gains of up to thirty points. So the paper offers two layers of improvement over a bare model: structure, in the form of tools and stages, and training, in the form of trajectories drawn from real projects.
Two lessons are drawn from this. Complex business intelligence workflows benefit from combining tool-augmented reasoning with domain-specific post- training, and the results point to promising directions for future research. The framing is what gives the benchmark its edge. It measures the task that users actually face, preparation included, and on that full task even frontier models stay below half.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every answer in business intelligence hides a pile of preparation. Here is that pile.
You work with business analysts who use Power BI and Tableau. Before they can answer a single business question, they must find the relevant tables, transform the data, and build join relationships by hand.
The road ahead
- 1 · The problem ~6 min Answering a business question is easy; preparing the raw tables for it, by finding, reshaping and joining them, is the hard part.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Split the BI workflow into search, join and transform subtasks, give the model specialized tools for each, and train it on trajectories synthesized from real projects.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Break the workflow into subtasks, expose data-management tools, synthesize traces from real projects, then post-train with supervised fine-tuning followed by reinforcement learning.
- 6 · The experiment ~6 min You’ll guess: Handed real business intelligence projects end to end and working in SQL, how accurately did even frontier LLMs answer the questions on BI-Bench?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.