Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data

Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data. By Milad Rezaei Hajidehi and two others, from Harvard University. Published on the thirty-first of August. No code repository is linked to it.

A great deal of valuable data still lives in unstructured sources: web pages, reports, contracts, filings, earnings calls and PDFs. The big bet in enterprise AI, as the authors frame it, is deploying LLM agents that reason over this data to answer complex questions for every knowledge worker. Agents can do that today. What stops them is cost. Each question sends the agent back into large documents to recover scattered evidence, and a single question can consume up to a million tokens.

Structure would change the arithmetic. If the data were already structured, the same question would reduce to a cheap database lookup. On the FanOutQA benchmark, reasoning over an ideal pre-structured store is twenty-eight times cheaper, and the gap grows to orders of magnitude as questions fan out over more documents. Yet structuring everything in advance is not viable. Documents hold vastly more possible structure than any workload will ever use, and which structure is useful, in which documents, stays unknown until the queries arrive.

The proposal is agentic data cracking: structuring unstructured data adaptively and speculatively, as a byproduct of reasoning itself. Adaptive, because the queries actually observed decide when structuring happens and what matters. Speculative, because it goes beyond the question currently being asked. The structuring is not a separate job run ahead of time. It rides along on the reasoning the agent was going to do anyway.

The mechanism attaches to a moment that already happens. Whenever the agent opens a document to answer, a cracking sub-agent forks from the context that is already loaded, which is why it costs only a marginal amount. That sub-agent extracts grounded structure likely to serve related future queries. Over time, an increasing share of queries is fully covered by structured data and can be answered without opening a document at all.

Notice what that does to the trade-off the problem set up. Structuring everything ahead of time was ruled out because nobody knows which structure will matter. Cracking never has to know in advance. It structures only documents that a real query has already caused the agent to open, and it extracts what is likely to serve related queries, so the store fills in along the lines the workload is actually taking. The aim is to keep agentic accuracy while bringing the cost close to that of RAG, the cheaper retrieval approach.

The test extends FanOutQA with merely one related question per test question, which is a small amount of the repetition that cracking is built to exploit. Even under that setting, cracking cuts cost by fifty-three percent while preserving accuracy. The problem was framed as a choice between agentic accuracy and a much cheaper bill. Here the bill falls by roughly half and the accuracy does not fall with it.

The authors present this as a first step toward next-generation data infrastructure for agentic reasoning over unstructured data. What they picture is a shared substrate beneath the model, where knowledge that reasoning has already paid to uncover accumulates rather than being spent once. The effort of reading a document is paid a single time, and the structure that reading produced stays behind for whoever asks next.