Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution

Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution. By Kyudan Jung and seven others, from Qualcomm AI Research and KAIST AI. Published on the sixth of October. No code repository is linked to it.

A speech assistant that uses tools usually runs its stages one after another. Automatic speech recognition first, then large language model inference, then external tool execution. The consequence is that tool latency only starts once the user has finished speaking and the model has worked out which tools to call. All the time the user spends talking is time in which nothing tool-related happens, and the wait for the tool is added on top of everything else.

The proposal is speculative tool execution for on-device cascaded voice agents. Instead of waiting for the full transcript, the system predicts tool requests from partial recognition hypotheses and starts executing the tools while speech is still arriving. The tool's wait is moved into the time the user is already speaking, which reduces end-to-end response latency.

A Predictor module anticipates tool calls during speech recognition, executes them speculatively, and caches the results. Those cached outputs are then injected into the model's prompt, so the response can come sooner. People correct themselves mid-sentence, though, and a speculative call made before a correction could be wrong. To handle that, a rule-based validation mechanism injects only the cached results that are valid. As a final safeguard, the model keeps the ability to issue tool calls directly. That guarantees the worst case: the framework's latency is upper-bounded by the baseline pipeline that runs everything in series. Speculation can help, but it cannot make things slower than not speculating.

The evaluation uses live measurements from a fully implemented Android voice assistant. Median time to first audio falls from five point seven nine seconds to four point six seconds. The standard deviation falls too, from three point four nine seconds to two point eight one seconds.

Two improvements come out of one change. The typical wait is shorter, and the spread of waits is narrower, so response latency becomes more predictable. For a voice assistant, that second number matters alongside the first: a reply that arrives at a consistent moment is easier to live with than one that is sometimes quick and sometimes slow. The design gains its safety from layering. Prediction starts the work early, validation filters out results that a self-correction has made stale, and the ordinary serial path is always still there underneath, so guessing early carries no worst-case penalty. The whole arrangement runs on the device, inside a cascade of recognition, language model, and tools that otherwise stays as it was.