Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks

Kierston & KierstonBench. Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks. By Andrew Semenov and one other. Published on the third of November.

Large language models are candidates for collecting and analyzing records in criminal investigations and in mass surveillance. How capable and how accurate they are at that work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark tests these capabilities in a simulated environment where the truth is known.

KierstonBench tests whether a model can investigate records and reach conclusions that match events that really happened in a simulation. Its thirty-two task families cover record linkage, reconstructing people's routines, investigating incidents, and assessing accounts that may be deceptive. Because the events are simulated, each one is known in advance, so a model's answer can be scored against what actually took place rather than against another analyst's opinion. That turns a question that is currently unanswerable, how accurate is this model as an investigator, into one that can be measured.

Underneath sits Kierston, a simulation engine at the scale of a city. It models about one hundred twenty-five thousand residents of Allentown, Pennsylvania, generating their daily activities along with the records that surveillance and investigative systems collect about them: telecommunications, financial, administrative and license-plate records. All of these derive from one common behavioral timeline, and they are spread across sources that stand for separate providers. Models investigate through a restricted records interface. They have to link observations across sources, and they never see the simulation underneath. In effect the model is placed where a human investigator would sit: in front of fragmented records held by different providers, with the full story available only by connecting them.

Sixteen language models were evaluated. Most could not perform records investigation at all, and submitted identifications that the records did not support. A small number of frontier models solved most of the tasks and abstained when the evidence was insufficient. Even those models, in some cases, still fell for common deceptions and assumptions, and none of them replaced an investigator.

The value of the setup is the ground truth. Because every record traces back to a simulated timeline, a model's conclusion can be checked against what actually happened, which is exactly what a real investigation cannot offer. On that footing the picture is split: most models make unsupported identifications, a few frontier models do the work and know when to hold back, and none is a substitute for the investigator.

Kierston & KierstonBench. Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks. By Andrew Semenov and one other. Published on the third of November. This is the long version: the whole paper rather than the abstract.

Suppose a language model is handed a phone number, a batch of plate reads and a bank feed, and asked who fired shots on a street corner twelve minutes ago. Is its answer right? On real investigative data nobody can say. Real deployments lack the ground truth needed to score a conclusion, and independently generated tables cannot show whether two records describe the same behavior. The authors' response is to build a city first and derive the evidence from it. Kierston simulates about one hundred and twenty-five thousand residents of Allentown, Pennsylvania, and KierstonBench asks models to investigate them. The authors say plainly that both are dual-use: the capability being measured is identifying and locating people from provider records.

The central design choice is the order of generation. Behavior comes first, records second. Every resident gets a daily timeline of work shifts, school runs, errands, gatherings and sleep. Only then are observations derived from it: payments at the venues visited, cellular signaling as a handset moves, calls and texts, and plate reads when a car passes a camera. Because everything flows from one timeline, records held by different providers genuinely agree, and the activities that produced them survive as an answer key. Random choices are keyed to the seed, the resident, the date and the activity type, so one person's day regenerates identically whatever order residents are processed in.

The model never sees that timeline. It works through a records interface split into fifteen sources, each standing for a separate holder: Social Security, the transportation department, county courts, banks, carriers, the camera operator. No person key is shared between them. A plate leads to a registration, the owner's SSN leads to the banks, a phone number leads to a carrier's subscriber accounts. The only joins are the identifiers inside the records, which is exactly an analyst's position.

Several details make the records hard in realistic ways. Social ties carry two independent weights. Exposure is how often a pair are together; affinity is how strong the relationship is. A colleague seen eight hours a day with little personal connection produces co-location without communication. A close friend living elsewhere produces communication without co-location. Exposure drives travel and co-presence, affinity drives calls, messages and payments. Carriers hold signaling events, not continuous position. Timing advance adds a distance from the antenna in steps of about seventy-eight meters, and call records carry only the serving cell, so finer location needs a join to network events. The seventy-four plate cameras misread in the ways optical character recognition does, zero for O, eight for B, and a misread can produce the plate of a different real vehicle. Confidence ranges overlap, so a high-confidence read can be wrong.

Incidents are staged as edits to a day already simulated. A seeded script binds roles, for instance a handgun owner with a criminal docket living near a strip of shops. The runner re-plans only the people the script touches and those affected as consequences, then regenerates their records, so the incident blends into the day. The shots-fired script also binds a decoy, an armed resident near the scene who is not the shooter, whenever the day contains one. Scoring runs from minus one to one. A correct identity earns credit by specificity, up to a full point for an SSN; a wrong one costs the same, and wrong entries in a ranked list are summed, so a long list of suspects is punished. Abstaining scores zero.

There are thirty-two task families, one task each: twenty-two Technical Tasks, seven Real-Time Investigations, three Deception Tasks. Sixteen models ran them. Routine linkage was largely solved: every model scored full marks on plate reconciliation and account resolution. The comparison that tests the thesis is the five suspect-identification incidents. No model identified a perpetrator in any of them. Every model that named someone was penalized. The logs show why. In shots-fired, Claude Fable five point one kept only handsets with a network event during the shooting window, and the shooter's handset had released its connection one minute earlier. That filter was never revisited. Suspects were ranked by firearm ownership and criminal records; in school-shooting, the staged perpetrator was a student who died at the scene. Deception went better. In disappearance, Fable matched a plate read of the caller's car and a handover of the missing woman's handset in the same minute near the same point, while his own handset stayed home.

The paper has a short limitations section, and it is candid. Each model ran every family once, on one sampled instance of one simulated day, because of cost, and the authors concede that small differences between models are within what a second attempt could change. The disappearance task was first staged with a flaw: the abandoned handset kept placing calls. Seven models' results on that task come from the flawed version. The behavioral analysis covers only some runs; those of Claude Opus five point five and Sonnet five point five are not analyzed. The city has known gaps too: one travel mode per person, no weather, no holidays, and messaging volume tuned to look plausible rather than fitted to measured data.

The idea to carry away is to simulate the world first and derive the evidence from it, so that an investigative conclusion can finally be scored against what actually happened. Measured that way, a few frontier models have the component skills: they resolve identities across providers, reconstruct a day, catch a false witness and abstain. Yet they still treat presence near an incident as participation, and a firearm license or a criminal record as evidence of involvement. In the authors' words, these models can assist an investigator. They cannot replace one.