CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse

CPU-ARC. Agentic Reasoning Caching for on-CPU Reasoning Reuse. By Peter Belcak and four others, from NVIDIA. Published on the sixth of November.

The authors open with a blunt complaint: large language models reinvent the wheel all the time. Their diagnosis is that a model conflates two different activities. One is solving a task, working out how it should be done. The other is executing that solution on a particular instance. Because the two are fused, the model is forced to solve every problem from scratch.

The claim is that the two can be pulled apart. Once a task has been reasoned through, applying its solution to a new instance of the problem can be done very efficiently, and with little involvement from a language model at all. The authors call the approach agentic reasoning caching.

What the method constructs are agentic reasoning caches, or ARCs. They are built from the reasoning traces of an LLM, and they are executable on a CPU. The abstract does not describe how the construction works. The picture to hold is simply that the model reasons once, and what it worked out is kept in a form a CPU can run.

The evaluation covers a range of difficult mathematical and scientific tasks. On those, ARCs perform on par with frontier LLMs. Several further properties are claimed for them. They are fully explainable. They are continually trainable. They are robust to perturbations of their input. And they are between sixteen and two hundred and ninety-one times cheaper to invoke than the source model they were built from.

The proposal reframes where the cost of reasoning should fall. Reasoning is paid for once, by the language model, and the result is reused many times on a CPU at a fraction of the price. On that view, the question for any repeated workload becomes whether it has already been reasoned through.