DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

DecisionRobust. Benchmarking Robustness for JEV Style Decision Models Across Text and Vision. By Yifan Lan and six others, from the Pennsylvania State University. Published on the thirteenth of October. No code repository is linked to it.

A new kind of model is emerging for decisions. Models such as JEV offer typed interfaces that take a task instruction and an observation and map them directly to a choice, a judgment, or a score. As these decision models are used more and more for automated decision-making, how reliable they are under adversarial manipulation becomes a critical question. That question needs a systematic way of asking it, across both the text and the images such models receive.

DecisionRobust is a benchmark built to answer it. It measures the robustness of decision models across textual and visual inputs, in practical decision-making scenarios. The attacks form a diverse suite: semantics-preserving perturbations of text, perturbations of images, and prompt injection, in both modalities.

Evaluation sets are curated through external validity checks and through cross-model candidate selection. That second step raises an obvious worry: an attack chosen on some models might only work on those models. So the evaluation deliberately includes held-out models. Thirty model configurations are evaluated on text tasks, of which twenty-four were excluded from candidate selection. Seventeen configurations are evaluated on vision tasks, of which twelve were held out.

Three findings are reported. The first is about transfer. Attacks expose vulnerabilities beyond the models used to build and select them. The exception is optimized image perturbations, which transfer weakly to many targets even when several surrogate models are used.

The second is that robustness is not one property. Which models are more robust changes across tasks and across attack types, and a model that resists input perturbations does not necessarily resist prompt injection. The third concerns measurement itself. Label-consistency checks substantially reduce the measured attack success, in both text and image settings. Some apparent failures, in other words, do not survive validation, and the authors stress the importance of checking them.

Two warnings sit inside these results, one for model builders and one for evaluators. A decision model can look robust under one kind of attack and fail under another, so no single test certifies it. And an attack success rate is only as trustworthy as the check that confirms each failure was real. The benchmark is publicly available.