Competence-Based Evaluation of Logical Reasoning in Large Language Models

Competence-Based Evaluation of Logical Reasoning in Large Language Models. By Jize Jiang and one other, from the University of Illinois Urbana-Champaign. Published on the sixteenth of October. No code repository is linked to it.

Large language models perform well across many tasks, yet their logical reasoning still has shortcomings, including inconsistency when the same problem is phrased differently. Improving reasoning requires measuring it accurately, and current benchmarks mostly report a single accuracy number on selected reasoning tasks. The authors argue that this performance-based methodology says little about whether a model's reasoning generalizes. A good score on a fixed set of items does not show that the model applies the underlying skill reliably, at larger sizes or in other wordings.

The proposed paradigm is competence-based evaluation, or CBE. It tests whether a model reliably applies the same competence across a whole family of problems, rather than whether it gets particular items right. Two ingredients make that testable. A parameterized generator scales complexity while holding the required competence fixed. And matched presentations that preserve the answer test correctness beyond any single formulation.

The paradigm is instantiated on elementary ordering tasks. Input size is scaled up, while the presentation of the same ordering rule is varied. Keeping the task elementary means the required competence stays plain, and what changes is only size and wording. Six models from the GPT and Qwen families are evaluated.

Performance is reasonably good over bounded ranges of complexity. But even strong models fail the competence condition across the tested family. Several interventions were tried. Chain-of-thought and GRPO extend the range of strong performance without resolving all of the tested failures. Supervised LoRA shows little transfer to held-out cases.

The contribution is a change in what a reasoning score is meant to certify. In place of a number on a sample, CBE produces diagnostic profiles of where a competence holds and where it gives way, together with an evaluation protocol that can grow as models improve. Ordering is an elementary task, and that is the point. A model that cannot keep one simple rule stable as problems grow and are rephrased has not yet shown the competence its accuracy might suggest.