What Git Does Not Record: Provenance Legibility in Two AI-Assisted Research Repositories

What Git Does Not Record. Provenance Legibility in Two AI-Assisted Research Repositories. By Koutian Wu and two others, from the University of Texas at Austin, the University of California, Los Angeles, and one other institution. Published on the ninth of October. Its code repository is public but slight: one source file.

Publishers and funders increasingly ask authors to disclose how AI tools contributed to their research. For computational work, the repository is where an author or an auditor would go looking for that answer. This study asks what a repository can actually supply, and its claim is that the record is partial in ways that depend on the kind of trace an agent leaves. Repository channels disagree with each other, and neither of the rates examined can be read as a rate of AI involvement.

The material is two single-investigator repositories, by the same author and in the same field. Each carries a different class of agent residue. One holds a prescriptive trace: standing rule files. The other holds a narrative trace: a running agent scratchpad. The repositories have four hundred and sixty-six commits and one hundred and three commits respectively.

The two traces are legible in opposite ways. Of all the lines ever committed to the prescriptive trace, eighty-five point six percent are still visible at the current revision, yet a standing rule states a conclusion and throws away the episode that led to it. The narrative trace does record episodes, and shows only twenty-one point eight percent. Counted by revision instead, the gap shrinks a great deal. Fifteen of the prescriptive trace's forty-one revisions leave no visible line, against eighteen of the narrative trace's forty-one.

Then the trailer. The Co-Authored-By trailer is the field an automated audit reads. In Project A it is one of three AI trailer keys, and it covers sixty-six of the one hundred and twenty-one commits that carry any of them.

In Project B it has low sensitivity measured against an explicit proxy inside the repository. Thirty-nine commits revise a file kept to hold an agent's working notes, and just one of them, two point six percent, carries the trailer. Project A's repository-level rate is unstable over time as well, stepping from two point seven percent to seventeen point seven percent across a boundary the authors did not specify in advance.

The study covers two repositories, both from one author in one field. The authors say plainly that they do not isolate the cause of the disagreement between channels. What they provide is a field-by-field account of which provenance claims such traces support, which they cannot support, and which they answer confidently and wrongly. They also construct an auditable evidence ledger.

From that account the authors derive requirements for any system meant to record research provenance as it happens. The people this speaks to are the publishers, funders and auditors asking for disclosure, along with the tool builders who would have to meet those requirements, since a commit history read after the fact is shown here to answer some of their questions wrongly.