Assessing the CAD Capabilities of General-Purpose AI Models: A Case Study of GPT-6 Astra

Assessing the CAD Capabilities of General-Purpose AI Models. A Case Study of GPT-6 Astra. By Qiang Zou and five others, from Zhejiang University. Published on the seventeenth of October. No code repository is linked to it.

General-purpose AI models and agentic systems have recently become able to perform some computer-aided design tasks. The authors argue that a systematic assessment is now needed: what can these systems accomplish, and how well do their outputs meet engineering requirements? Their answer, from a study of one state-of-the-art model, GPT-6 Astra, is that general-purpose AI is beginning to offer practical value in almost every step of a CAD workflow, but that a gap remains between geometry that looks right and models an engineer can use.

The study covers four basic CAD tasks: understanding a model, generating one, editing one, and evaluating one. Test cases span different levels of complexity and of engineering requirement. GPT-6 Astra can support multiple stages of the workflow, and it succeeds in cases involving nontrivial geometric complexity.

The failures are just as specific. Generating visually plausible geometry is not the same as producing a CAD model that is valid, functional and editable, and the study finds recurring errors in three places: freeform geometry, parameter dependencies, and the mating relationships between parts. Those errors are what limit engineering usefulness. Agentic iterative refinement, letting the system revise its own work, can improve results, but it does not consistently resolve the critical errors.

This is a case study of a single model, GPT-6 Astra, on four basic tasks and a designed set of test cases. Its positive conclusion is phrased as a beginning, value in almost all steps rather than a finished capability. And there is a practical constraint beyond correctness: long completion times further limit how usable the system is in real work.

The picture is of a model that can take part in nearly every stage of design while still falling short where engineering is strictest. Plausible shapes come readily. Correct dependencies, freeform surfaces and parts that fit together do not yet, and iteration alone does not close that gap.

Assessing the CAD Capabilities of General-Purpose AI Models. A Case Study of GPT-6 Astra. By Qiang Zou and five others, from Zhejiang University. Published on the seventeenth of October. This is the long version: the whole paper rather than the abstract.

A CAD model is not a picture of a part. It carries design intent, and it has to survive what comes next: editing, analysis, manufacturing. A model can reproduce a part's visible appearance and still contain passages that do not connect, or assembly interfaces that do not fit. It can match the starting geometry and then lose a required feature the moment one dimension changes. Those are exactly the failures a rendering hides. Agents can now run modeling code and revise what they built, so the authors, at Zhejiang University, test one of them, GPT-6 Astra, the way an engineer would. Earlier evaluations, they argue, cover subsets of CAD work with mostly synthetic cases. Here every case comes from real engineering practice.

Two representations carry the whole study. The first is the boundary representation, or B-rep: a solid described by its faces, edges and how they connect, exchanged here as STEP files. A B-rep tells you the shape. The second is the parametric feature model, which records a construction history of sketches, extrusions, holes and cuts tied together by dimensions, so that changing one number updates everything that depends on it. Between them sits solid validity: whether the result is a genuine closed solid, free of self-intersections and non-manifold connections. The study keeps four qualities apart: geometric accuracy, solid validity, functional adequacy and parametric editability. A model can pass one and fail another.

The assessment covers four basic tasks: understanding, generation, editing and evaluation, split into six kinds of test. Understanding means classifying twelve parts from STEP files, and naming the machining features in six B-rep models without being given a list of feature types. Generation means building models from engineering drawings, from twelve text descriptions, half geometric and half functional, and from rendered images with no dimensions. Editing means rebuilding eleven STEP models as native parametric models through the SOLIDWORKS interface, then changing their dimensions by hand. Evaluation is not a separate test. It is the agent's own checking and refinement, watched across all the others.

The agent runs at its maximum reasoning setting in the ChatGPT client's Codex mode. Each case gets a fresh conversation, memory off, and a single turn, inside which the agent may loop through building, checking and revising on its own, choosing its own tools. Web search is on by default. One detail governs every result: the model is tested together with its tools, and the experiments do not separate the two.

Verification is independent of the agent's self-assessment: solid validity checks, geometric comparison, and item-by-item inspection against the requirements. For images, agreement is measured two ways: silhouette overlap with the input views, and Hausdorff distance, the largest sampled gap between generated and reference surfaces, with no rescaling allowed. Valid parametric models receive ten single-parameter edits each. An edit passes only if the change takes effect, dependent geometry updates, the result stays valid, and the model can be restored.

Understanding is strong. All twelve parts went to the right class. Feature recognition was fully right in only three of six cases: the recurring error was labelling groups of faces as features when they were not, and in one complex model, merging two separate holes into one and calling an outside cylinder a hole. Generation has real successes: from detailed text, a V8 engine block as one valid solid, its water jacket kept apart from the cylinder bores.

Generation also shows the gap between scoring and working. All twelve text cases met every listed requirement, yet the authors insist a full score does not establish function. One gripper's motor shaft sits in its pinion with a radial gap and nothing to secure it, so it cannot transmit torque. Image reconstruction shows the same split. The case with the best silhouette overlap, zero point nine nine one five out of one, had a Hausdorff distance of about sixty-four millimetres.

The comparison that most directly tests the thesis is editing. Among the six models valid enough to edit, fifty-two of sixty edits passed, which is eighty-six point seven percent. Counted against all eleven cases, though, only four survived both the validity check and every edit: thirty-six point four percent. In one housing, deepening a side section closed an existing hole opening. Missing input cannot explain this. The STEP files contain the complete target geometry, and five of the eleven reconstructions still failed validity.

There is no dedicated limitations section, but the closing page of the full report states the limits plainly. The evidence covers selected cases and one system configuration. Complexity is defined separately for each task, so the levels are not a common scale. The supplementary comparisons are few and do not measure variability across repeated runs. Embedded STEP metadata may leak prior information: in the coupling case the agent cited the model name stored in the file, and the perfect classification score holds only within the supplied candidate list. With the list and web search both removed, that coupling was called a check valve. Editing was tested one parameter at a time; combined changes and full assembly function were not evaluated. Time is the last limit. Complex text cases averaged over an hour and a half, and one image reconstruction ran three hours and forty-one minutes.

The idea to carry away is that resemblance is not engineering. This agent can read parts, make design choices, and assemble mechanisms with internal passages. What it cannot yet do reliably is keep the relationships that make a model useful: passages that connect, parts that are secured, features that move when the dimension they depend on moves. Hence the authors' advice: use it as an assistant for initial models and design alternatives, verify geometry, validity, connections and dependencies yourself, and treat correction that does not break those relationships as the central open problem for CAD agents.