SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

SimuVerity. Benchmarking Agents for Engineering-Grade Simulink Model Generation. By Ruiqi Zhang and eleven others, from Xi'an Jiaotong University. Published on the first of October. Its code repository is worked out: about four thousand five hundred source files, with a readme and a licence.

Existing Simulink benchmarks mostly ask one of three questions about a generated model. Does it compile? Does it execute? Does it resemble a reference model? Each of those can come back yes while the real question goes unanswered, which is whether the model satisfies its engineering requirements. A model can run cleanly, look like the reference, and still not do the job it was built for. That gap between running and qualifying is what this benchmark is built to measure.

SimuVerity asks agents to turn text into an executable Simulink model, and then judges the result as an engineer would. The question is not whether the output looks right but whether it behaves right under the conditions its specification describes.

There are one hundred and one model-generation tasks, spread across ten engineering domains. For each task, executable-system profiles ground the engineering specification, along with four families of native simulation scenarios. Scoring is hierarchical. A model first has to clear three gates: the artifact must be delivered, it must be natively executable, and it must pass engineering qualification. Only qualified models go on to be scored, and they are scored across six dimensions: accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response.

Six agent systems were evaluated. The best of them reaches an overall score of only forty-two point eight six. The results show that structural similarity is a poor proxy for engineering performance. The bottlenecks come at two points rather than one. Some systems struggle to produce a qualified implementation in the first place, and even after qualification, satisfying requirements along many dimensions at once remains hard. A further observation cuts the other way: some models that score highly still show severe disorder in their visual layout.

A benchmark like this changes what passing means. Resemblance to a reference was a convenient stand-in, and the results here say it was standing in for the wrong thing. By separating delivery, executability and qualification from the graded dimensions that follow, SimuVerity offers a systematic basis for assessing what agents can actually engineer, and for locating where in that chain they fail.