WMIP: A Working-Memory-Inspired Programmable Probe of Cumulative State Tracking in Large Language Models

WMIP. A Working-Memory-Inspired Programmable Probe of Cumulative State Tracking in Large Language Models. By Dengzhe Hou and five others, from Tohoku University, the Lappeenranta-Lahti University of Technology, and two other institutions. Published on the thirtieth of October.

People rely on working memory to hold a piece of information and keep updating it. Large language models meet a related demand whenever a single prompt asks them to follow a series of updates without writing out the intermediate states. Measuring how well they do that has been uneven, because existing tests differ in what they allow about written steps. One test lets a model work things out on the page, another forbids it, and the two scores are not measuring quite the same thing.

The probe targets cumulative state tracking: keeping a value correct as change after change is applied to it. Answers are computed by a program rather than written by hand. Questions come in three kinds. A model may be asked for the final state, for some intermediate state along the way, or for several targets at once.

Depth is the main dial. An item can carry as few as three updates or as many as one hundred. Crucially, different output protocols are compared on identical items, so a change in score can be traced to the rules about writing, not to a change in the questions. The items and the model responses are released for reproducible comparisons across depths, query types and protocols.

Eighteen language models were tested. On single-step arithmetic they reach eighty-four point nine percent accuracy. On the core items, which need one number after three to seven updates, accuracy falls to thirty-one point seven percent. From fifty updates onward, no model exceeds ten percent.

Protocol moves the picture a great deal. Given a larger budget and instructions to write out steps and put the answer on the last line, core accuracy climbs to eighty-eight percent, and the gain comes mainly from the instruction changes rather than from the budget. Core accuracy also correlates positively with public scores on the Berkeley Function Calling Leaderboard.

Two lessons sit side by side. Being good at one arithmetic step does not make a model reliable at carrying a running total through many of them. And the rules for how an answer must be written can shift measured accuracy so far that comparing scores across tests without matching protocols tells you little.