Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Co-Design Bench. Multimodal Collaborative Coding Evaluation for Agentic Models. By Xilin Wei and fifteen others, from Fudan University, Alibaba Group, and four other institutions. Published on the twenty-sixth of October.

Evaluating human and AI co-design means testing something specific: whether coding agents can take real users' requests, which are often multimodal, and turn them into working revisions of an application. A user might attach a picture, a recording or a document, and the agent has to change running software to match. Testing that well calls for tasks drawn from what real users actually ask, and for grading that does not rely on a reference solution or a target screenshot to compare against.

Co-Design Bench is a reference-free benchmark of two hundred and seventy-three tasks, split into two tracks. Multimodal Editing covers localized edits, which may come with an image, audio, video or document as input. Repository Editing covers coordinated revisions that reach across a repository. Cutting across both tracks are three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation.

The tasks were curated from thirty-seven thousand four hundred and sixty-three deduplicated queries, which were themselves drawn from twenty-six thousand three hundred and fourteen real user coding sessions. Each task pairs a normalized request with a runnable application in its pre-edit state, and with hidden acceptance criteria written by experts. There are two thousand three hundred and thirty-one of those criteria in all.

Grading follows what the authors call a judge-then-refute protocol. First, criteria are graded using evidence from the source and from the browser. Then any positive verdict is challenged by a separate LLM verifier, so a pass has to survive an attempt to overturn it. Claims about visuals and interaction are audited too, all without gold code or target screenshots.

Ten model endpoints were run under a shared agentic harness. The strongest passes sixty-nine point nine percent of criteria per task on average, yet fully satisfies only twenty-six point four percent of tasks.

Difficulty also tracks the breadth of a change. All ten endpoints complete fewer criteria on repository edits that span three or four change types than on those spanning one or two.

The distance between those two headline numbers is the point of the benchmark. An agent can meet most of what a user asked for and still leave the revision unfinished, and Co-Design Bench is offered as a testbed for studying exactly that gap, between partial satisfaction of requirements and complete success on real-user application revisions.