LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

LiveMACE. Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets. By Jun Zhao and eleven others, from the National University of Singapore, Fudan University, and one other institution. Published on the seventh of October. Its code repository is worked out: about one thousand source files, with a readme and a licence.

Judging an agent only by its outcomes can hide the capabilities that produced those outcomes. That problem sharpens in environments that keep changing, because there the outcome is a closed loop. It reflects the agent's behaviour and the shifting external conditions together, and one cannot be read off without the other. A good result might come from a capable agent, or from favourable conditions; a poor one might hide a capability that was used well. A score on its own cannot tell these apart.

LiveMACEBench is built to look past the outcome. It is a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Agents are assessed in two ways at once: by realized outcomes, and by diagnostics specific to each mechanism, derived from their complete decision traces.

Five frontier LLMs run along continuous trajectories. Their configurations are matched across four mechanisms: Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration. Matching the configurations is what lets differences between models be attributed to how they use a mechanism, not to which mechanisms they were given. Because the agents are persistent and the market keeps moving, every decision leaves a trace, and those complete traces are the raw material for the diagnostics. The evaluation ran live for thirty days.

The headline is an outcome-capability gap. Realized returns often diverge from the capability-specific measurements, and similar outcomes can come out of markedly different patterns of mechanism use. Trace-level diagnostics go further, exposing distinct bottlenecks across the capabilities. The authors conclude that having access to a mechanism, using it effectively, and performing well downstream are three different things, and none can stand in for the others as a measure of agent capability.

A market is an obvious leaderboard: returns go up or down, and models get ranked. This benchmark keeps the market but changes the question asked of it, treating the live environment as a diagnostic instrument for what an agent is actually doing.