Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

Beyond Compilation. Evaluating Faithful Natural-Language-to-Lean Statement Formalization. By Ke Zhang and five others, from the University of California, Riverside, the University of Arizona, and one other institution. Published on the fifteenth of October. Its code repository is public and real: about forty source files.

Lean will tell you that a generated declaration is well typed. It cannot tell you whether that declaration states what the user meant. In statement autoformalization, where natural language is turned into a Lean statement and there is often no canonical Lean target to compare against, compiling is easy to mistake for being right. The question here is how far compilation overstates faithfulness.

The yardstick is a stricter acceptance criterion. An output must compile in Lean, and two LLM judges from different model families must both agree that it is faithful. On an independent random audit, this criterion agrees with the human majority on ninety-one point five percent of cases. Its errors lean one way: it rarely accepts an unfaithful output, but it does reject some faithful ones.

The test set is two hundred and twenty-seven university-level statements, and eight systems are run on it: seven one-shot systems and the authors' own tool-augmented agent. For every system, the measured quantity is the gap between the share of outputs that compile and the share that are accepted. Rejections are then reviewed to separate real translation errors from judge errors. The agent's rejections all receive human review. For the one-shot systems, a two-stage review by Claude Opus five point five working with Lean and GPT six point one Sol, validated on those human labels, does the sorting.

The gap ranges from one point three to twenty-nine point five percentage points across systems. The agent is the clearest case. It compiles eighty-seven point two percent of the statements, but only fifty-seven point seven percent are accepted. Human review confirms that twenty-three point eight of its twenty-nine point five points are real translation errors. For six of the seven one-shot systems, the review finds that between seventy-eight and one hundred percent of the gap is real translation error.

Then a twist about the judge itself. GPT five point two, one of the two judges, grades two models released after it: GPT six Astra and Claude Opus five point five. It accepts eighty-three percent of statements for each, against twenty-one percent for one-shot GPT five point two. Yet the review classifies only four of their fifty-four rejections, about seven percent, as translation errors. Most of their measured gap is the judge's error, not theirs. That fits a simple pattern: the fewer errors a system makes, the smaller the share of its rejections that are real.

Finally, an ablation over three tools, two settings each, shows that Lean feedback drives most of the agent's compilation gain, and that nearly half of that gain fails the semantic criterion.

Three lessons stack up. Compilation can overstate faithfulness by tens of points. A good judge is still a fixed judge, and as systems improve, a growing share of what it rejects is its own mistake. And of the compilation gained from Lean feedback, nearly half buys a statement that compiles without saying what was asked.