TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

TAC-CoT. Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens. By Juseon Do and three others, from KAIST and Cluvion. Published on the first of November.

Agentic systems built on large language models are more and more often paired with chain-of-thought reasoning, for tasks that need accurate reasoning, external knowledge and deliberation over several steps. The two available styles of chain of thought each fall short here. Explicit chain of thought, written out as text, carries a heavy generation overhead. Implicit chain of thought swaps the written rationale for compact continuous tokens, which is cheaper, but it does not naturally provide the explicit communication interface that tool use requires. A tool call has to be spelled out, and a latent thought is not spelled out.

TAC-CoT is a tool-empowered, adaptive implicit chain-of-thought framework that keeps text only where text is unavoidable. The reasoning traces become compact latent segments. Textual tokens remain for three things: tool calls, the observations that come back from tools, and final answers.

The result is an interleaved sequence. Latent reasoning runs between the points where the agent has to talk to a tool or to the user, and explicit tokens appear at exactly those points. That lets the agent use tools without putting every intermediate reasoning step into words.

On agentic math and question answering benchmarks, TAC-CoT improves accuracy by up to eight point eight points while cutting the number of model forward passes by up to six point eight times. Further analyses show better tool-use quality, both when a single tool is available and when several are.

The design rests on a simple division. Reasoning that nobody outside the model has to read can stay latent and cheap, and the parts an external tool or a user must read stay in text. Both numbers move in the right direction at once: higher accuracy, and far fewer forward passes.

TAC-CoT. Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens. By Juseon Do and three others, from KAIST and Cluvion. Published on the first of November. This is the long version: the whole paper rather than the abstract.

Agents that use tools usually think out loud. Before each search or calculator call, the model writes a passage of reasoning, reads the result, and writes another. Every word costs a forward pass, and on multi-hop question answering the explicit agents here average well over two hundred and forty forward passes per query. Long written traces also accumulate errors: every step is one more place to go wrong. Implicit chain of thought is the known alternative: the model reasons in continuous vectors that are fed back as inputs and never decoded into words. But existing implicit methods spend a fixed block of those vectors in one uninterrupted run before the answer. There is no point at which to emit a tool call or read an observation, and every query gets the same budget however hard it is.

Two ideas need to be in place. The first is a continuous thought token. Instead of picking a word, the model takes its last-layer hidden state, passes it through a small projection layer, and feeds the result straight back in as the next input embedding. No vocabulary index is ever assigned. The second is the explicit agent being replaced. Its trajectory runs: a reasoning segment, a tool call, an observation, another reasoning segment, and so on, ending with a segment that makes no call and consolidates what was gathered. TAC-CoT keeps that skeleton and swaps only the reasoning segments for short runs of continuous tokens. Tool calls, observations and answers stay as ordinary text, because a calculator or a search engine can only read text.

At inference the loop is short. The model generates a segment of continuous tokens, three by default. Then it compares two logits: one for taking an action, one for answering. If the action logit is higher, it decodes a tool call as text, the executor runs it, the observation is read in, and another latent segment begins. If the answer logit wins, the current segment becomes the consolidating one and the answer is decoded.

Training needs explicit trajectories with tool calls in them. For math, two existing GSM8K augmentations share their questions, one holding only the equations and the other only prose, so aligning them inserts calculator calls into the prose. For question answering, DeepSeek-V3 solved HotpotQA questions as a ReAct-style Wikipedia agent, keeping only correct trajectories: forty thousand of them, for about a hundred and three dollars.

The objective is self-distillation: the same backbone plays student and teacher. The teacher trains with ordinary cross-entropy on the full explicit trajectory. The student replaces each reasoning segment with continuous tokens, and an auxiliary decoder, discarded at inference, must reconstruct the hidden reasoning from them: each segment is cut into as many chunks as there are tokens, and each token must reconstruct its own chunk. That forces the latent segment to carry what would have been written there. Finally, at the colon just before the answer, the student's hidden activations at every layer are pulled toward the teacher's. The student, after reasoning silently, should arrive where the teacher arrives after reasoning aloud.

The backbones are small: a half-billion-parameter Qwen2.5, a one-billion parameter Llama, and a four-billion-parameter Qwen3. Averaged over all three, TAC-CoT beats the explicit tool-using agent, trained on the same trajectories, by five point eight points on math and two point four on question answering. The largest single gain is eight point eight points on GSM8K with the smallest model. Forward passes fall by up to two point six times on math and six point eight times on question answering, where trajectories are longer.

The comparison that tests the thesis most directly is not in the abstract. Training the smallest model, the authors make explicit segments latent one at a time, up to four. Average math accuracy climbs at every step, from forty-one point one to forty-five point four, while forward passes more than halve. If writing the reasoning down were what kept it reliable, going silent should hurt. It helps, monotonically. A second analysis tests the budget argument. Against SIM-CoT, a fixed-budget implicit method, the lead on GSM8K grows from four point six points on one-step problems to thirty point three on four-step ones.

The paper's limitations section is two sentences long: routing happens per segment rather than per token, and longer interaction horizons remain to be evaluated. The evidence suggests more caution. The six point eight times saving is counted in forward passes; measured on a single NVIDIA L40S GPU, wall-clock speedups are two point three times on math and two point four on question answering, and the authors warn against equating the two.

Each question-answering benchmark uses a thousand sampled examples, for which the paper computes a worst-case margin of about three point one points, larger than the average gain there, though the main comparisons are tested with a paired bootstrap. On ToolBench, judged by GPT-5.1, the single-tool win rate is fifty-one point eight percent, close to a tie; only the multi-tool result is significant. And the reasoning is no longer readable: the judge had to be told that TAC-CoT does not verbalize it.

The idea to carry away is a division of labour inside one trajectory. Whatever must cross to the outside world, a tool call, an observation, an answer, stays in words. Everything else can stay latent. Cut the reasoning at those boundaries, and two problems of implicit chain of thought dissolve together: tools become callable, and the budget grows with the decisions a query needs. In the paper's overtime-pay example, the explicit agent applied the overtime rate to all forty-five hours. TAC-CoT wrote nothing but four calculator calls and the right answer, four hundred and sixty dollars.