BIABench: Evaluating AI agents on real-world bioimage analysis tasks

BIABench. Evaluating AI agents on real-world bioimage analysis tasks. By Zixuan Pan and seven others, from the University of Notre Dame, the Leibniz Institute for Analytical Sciences, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about ninety source files, with a readme and a licence.

Artificial-intelligence agents are a promising way to automate bioimage analysis. No benchmark, though, has tested whether they can carry out a real-world analysis from end to end. That is hard for an agent for a concrete reason. Two-dimensional images, three-dimensional volumes and time-lapse sequences are often too large to read in as context. So the agent cannot just look. It has to choose an analysis and run it, through code, specialized software and rendered views. Whatever the agent learns about the data, it learns through tools it picked and steps it chose to take. That is precisely the capability a benchmark would need to test, and the one no existing benchmark tests end to end.

Published studies are what make this testable, because each one pairs raw images with a peer-reviewed result. The study supplies both the question and an answer that has already been checked by reviewers. BIABench is built from sixteen tasks reconstructed from published biological studies, each keeping its scientific question, its imaging data, and its ground truth. Together the tasks span eleven analysis subtasks, and modalities running from H and E histology to single-molecule localization microscopy.

Each submission gets two scores. The outcome score compares the output files against the ground truth, using metrics that are standard in the field. The process score has a vision-language model judge the agent's choice of method and its quality control, against a rubric written by experts. The authors evaluated both general-purpose and biology-specific agents, across several language models, and ran every task more than once. The two scores ask different things. One asks whether the answer was right. The other asks whether the work was done the way an expert would do it.

Routine two-dimensional tasks were handled well. Add a third dimension or a time axis, and the picture changes: on some of those tasks, no agent scored above zero point one nine. Nothing tried closed that gap. Not biological specialization, not stronger models, and not detailed instructions written by experts. Each of those is a natural lever to pull. On the hardest three-dimensional and time-lapse tasks, none of them was enough.

Reliability was its own problem. Scores varied more between repeated runs of one agent than between different agents. And without the ground truth, a correct run could not be told apart from a wrong one, either by its process score or by how long it took. Put plainly, the noise inside one agent was bigger than the gaps between agents, which makes any single run a weak basis for ranking them.

That last result is the sharpest. Neither the quality of an agent's process nor the time it spent tells a right analysis from a wrong one, so only the ground truth can, which is exactly what a benchmark built on published results supplies. BIABench is released openly with its data and code, as a verifiable framework for evaluating these agents, and eventually for training them toward reliable, long-horizon bioimage analysis.