MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

MMEnterprise-Bench. Evaluating Enterprise Agents over Evolving Multimodal Records. By Seohyun Lee and six others, from KAIST and Korea University. Published on the twentieth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

An enterprise agent answers questions over workplace records, and those records evolve as a team collaborates. Information is shared, then corrected, then revised, across documents and chats and figures, until several versions of the same fact sit in the workspace together. Answering reliably in that setting asks three things of an agent. It must reason over sufficient evidence. It must resolve conflicts between sources and across time. And it must recognise when the evidence available simply is not enough to settle an answer.

MMEnterprise-Bench is built to evaluate those capabilities systematically. It holds one thousand six hundred and thirteen questions, and they fall into three categories that match the three demands: Evidence Reasoning, Conflict Resolution, and Unanswerability. The third is the one that asks an agent to decline rather than to answer.

The questions are grounded in evolving multimodal records drawn from simulated collaboration projects. Simulation is what makes the evolution available: a project whose documents, chats and figures accumulate revisions gives the benchmark the superseded and conflicting versions it needs, each question anchored to the record as it stood.

Ten multimodal LLMs are evaluated, both proprietary and open-source. Even the strongest of them struggle in three situations: when evidence carries conflicts between sources, when it carries conflicts across time, and when it spans multiple images. A second result concerns retrieval. Increasing the retrieval budget, or using a larger retriever, does improve evidence recall, but that improvement does not consistently translate into higher answer accuracy.

Finding the right evidence and using it turn out to be separable problems, and the retrieval result is where that shows. More recall, no reliable gain in accuracy. The authors take their findings to highlight how persistent the challenge of reasoning over evolving workplace records is, and offer the benchmark as a testbed for building enterprise agents that are more reliable. Code and data are available.