FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

FINCOSMOS. An Open-World Evaluation Paradigm for Agents in Finance. By Yupeng Cao and sixteen others, from The Fin AI, the Stevens Institute of Technology, and five other institutions. Published on the fifth of November.

Language agents increasingly act for users on the open Web, and the authors argue that finance is where trusting them is hardest. Three things make it so. The data has to come from the Web. The strategy lives in the user's head. And correctness depends on time. Together these let an agent fail silently. It might pursue a strategy the user never asked for, rely on data nobody verified, or run a backtest that peeks at the future, and the resulting report can look exactly as convincing as a correct one. Existing benchmarks, by this account, close the world somewhere. They fix the task, or the data, or the timeline, and in doing so they remove these failures rather than measure them.

FINCOSMOS is an open-world evaluation paradigm for agents doing financial work. What it tests is whether an agent can do the whole job with none of those pieces handed to it: find out what the user wants, get the data itself, respect time, and hand back results that can be checked.

A simulated user holds a hidden strategy, sampled from a compositional grammar, and reveals it only when asked. The agent receives no data at all. It has to acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results.

Grading needs no reference answer. Programs verify point-in-time correctness and recompute every metric the agent reports. The delivered strategy is checked against the hidden specification. A run is accepted only when the simulated user is satisfied and none of fifteen veto checks fails. Satisfying the user is necessary, but it is never enough on its own.

The evaluation spans four coding agents and a model without tools, across one hundred and fifty sessions. Not one session is accepted. Every session that the simulated user signed off on fails a program check. Agents raise only fifty-four to sixty-four percent of the details they were not told, which means a good share of the hidden strategy never comes up.

Human experts reviewing the failures attribute eighty-eight point three three percent of them to the agents. One more result points the other way. Given FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.

A clean zero is the headline, and its source matters. Every session the simulated user was happy with still failed when a program rechecked the work, which is precisely the silent failure the design was built to expose. User approval and correctness came apart in every case. The memory result suggests the same grading that rejects everything can also serve as a teacher.