StudentBench: AI and human tutoring yield equivalent GRE learning gains

StudentBench. AI and human tutoring yield equivalent GRE learning gains. By Curtis Northcutt and five others. Published on the twenty-third of September. Its code repository is worked out: about ninety-five source files, with a readme and a licence.

The headline claim is in the title: tutoring by AI and tutoring by humans yield equivalent GRE learning gains. The authors establish that AI tutoring is statistically equivalent to expert human tutoring for GRE learning gains, with a p value of zero point zero one five. In five of the seven GRE domains, the best- performing AI tutor surpassed the human tutor, on average. The human side of that comparison is expert human tutoring.

The evidence comes from StudentBench, a suite of AI teaching evaluations and a public platform for large-scale data collection, holding over one hundred seventy-five thousand messages between students and AI. With it, the authors measured learning gains on quantitative and verbal GRE questions across two thousand three hundred eighty-three human participants, who received AI tutoring, human tutoring, or no tutoring. The group without tutoring gives the baseline against which the gains from either kind of tutor are measured.

In a second study, expert human tutors compared lesson plans and practice problems generated by language models, through two thousand twenty-eight pairwise rubric evaluations. Together, the two studies separate AI tutors on five dimensions: lesson planning, practice-problem creation, conversational pedagogy, cost, and engagement. The first study measures what students gain. The second looks at the materials a tutor prepares, as judged by human experts.

One AI tutor achieved learning gains equivalent to human tutoring, with a p value of zero point zero four four, at nine hundred eighteen times lower cost. Per percentage point gained, that is zero point zero zero five two dollars for the AI, against four point eight one dollars for the human. For quantitative GRE sessions, faster AI replies correlated with more student messages, more messages with more correct practice, and more correct practice with larger learning gains, each with a p value below zero point zero zero two. Read as a chain, it links something a system designer controls, the speed of a reply, to the outcome that matters, how much a student learns.

The claims are bounded by what was measured. Equivalence is shown for learning gains on quantitative and verbal GRE questions. The surpassing result concerns the best AI tutor, in five of seven domains, on average. The cost figure is per percentage point gained, for one particular AI tutor, and the reply-speed findings are correlations within quantitative sessions. They show that these things moved together, not that faster replies caused the gains. And the setting throughout is one exam, the GRE, rather than learning in general.

The authors frame the work against a frontier focused mainly on advancing model capabilities. StudentBench points the measurement at people instead, at whether they learn, and the platform is freely available.