Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systems

Dream2Discern. Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systems. By Boxuan Zhang and three others, from Rutgers University, the University of Texas at Austin, and one other institution. Published on the twelfth of October. Its code repository is public but slight: eight source files.

Reliable deployment of LLM-based multi-agent systems calls for runtime auditors: models that watch a run as it unfolds and predict, from the prefix of execution seen so far, how it will end. Such auditors usually learn from a limited corpus of trajectories whose outcomes have been verified. Growing that corpus is expensive, because every new trajectory has to be executed and then verified. And even broader coverage of failures does not, by itself, say which prefixes the current auditor actually finds hard to classify. More data, in other words, is both costly and untargeted. Gathering it gives no explicit criterion for which examples would teach the auditor the most.

Dream2Discern is a runtime auditor trained by implicit augmentation, and the augmentation is steered by the auditor's own prediction loss. Rather than collecting more examples, it manufactures difficult ones from the examples it already has, choosing them precisely because the auditor gets them wrong.

Training alternates between two phases that give the method its name. In the Dream phase, perturbations of the auditor's representation operators are selected to increase its prediction loss on recorded prefixes. What comes out are hard latent variants: versions of recorded prefixes, made in the auditor's representation rather than in new runs, that the auditor struggles with. In the Discern phase, the auditor learns to predict execution outcomes from those variants, using the original labels. Because the variants are regenerated as the auditor improves, the hard examples keep pace with it. A limited corpus becomes a renewable source of difficulty, with no additional trajectory collection and no extra outcome verification.

Evaluation uses two benchmarks, AFTraj-2K and StepShield. On both, Dream2Discern outperforms competitive failure auditors, and achieves over ninety percent trajectory-level weighted accuracy. It does so while balancing two aims that pull against each other in runtime auditing: being accurate, and being timely.

The scarce resource here is verified trajectories, and the method spends none of it. Difficulty is generated inside the auditor's own representations, labelled with outcomes already known, and refreshed every time the auditor catches up.