MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification

MotionGraphicsBench. Evaluating Motion Graphics Generation with Programmatic Verification. By Jiaju Ma and three others, from Stanford University and Figma. Published on the seventh of November.

Large language models can already produce motion graphics, starting from a static image and a request written in natural language. Judging how well they do it is the harder problem. A real request tends to combine several kinds of requirement at once: where things are in space, how they look, and how they move over time. Checking all three together, for one animation, is where evaluation has struggled.

MotionGraphicsBench is an expert-curated set of one hundred instances. Each one hands the model a static SVG and a motion edit prompt, and asks for an animation of that graphic. Every required visual element is supplied up front. That choice mirrors how a designer might work, adding motion to artwork that already exists, and it isolates motion generation from graphic generation, so a model is not judged on drawing at all. The instances derive from real-world Lottie motion designs, and they cover diverse combinations of spatio-temporal animation behaviour.

Grading is where the benchmark departs from the usual routes. Each instance comes paired with a verification program: deterministic, readable by a person, and able to accept different valid implementations of the same request. It reports which of the prompt's requirements pass and which fail. There is no comparison against a single reference animation, and no vision-language model acting as judge. The programs themselves were built by a multi-agent pipeline, working from human-labelled animations and from synthesized test cases.

Whether such programs can be trusted was tested directly. On a balanced test set of one hundred eighty animations, they agree with human judgments ninety three point three percent of the time. A GPT-6 Astra vision-language baseline reaches eighty three point three percent, and the programs also beat it on recall for both approvals and rejections.

The headline sweep covers forty-seven configurations, drawn from thirty-two language models across twelve providers. Their pass at one scores run from six percent at the bottom to sixty-seven point seven percent at the top.

A second sweep gives fifteen models their highest available reasoning effort and thirty-two attempts per instance. More attempts help every model. Even so, pass at thirty-two still spans thirty-eight percent to ninety-five percent. Reading the verification programs' reports shows where the generated animations go wrong. They often fail to match reference poses, fail to follow the requested motion trajectories, or fail to coordinate events that are meant to be related.

Because each verdict is itemised by requirement, a failure here arrives with a reason attached rather than as a bare score. A pose missed, a path not followed, two events out of step. Even the strongest model, given thirty-two tries, leaves room at the top, and the benchmark itself has been released so others can run it.