How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus. By Ilya Koziev and two others. Published on the fourteenth of September. No code repository is linked to it.

Orthrus is a hybrid autoregressive-diffusion architecture. It accelerates autoregressive language-model inference by generating multiple tokens in parallel, while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism makes this speculative decoding lossless, producing the same output sequence as the autoregressive model would. The finding here is that in practice this depends on numerical precision. Lossless is a strong word. It means not approximately the same text, but exactly the same sequence.

The authors independently reproduce Orthrus and examine the claim under different numerical precisions. Under BF16 inference, the trajectories match exactly in only forty-five percent of cases for the original authors' checkpoint, and in forty-three percent for the independently trained model, across one thousand one hundred ninety prompts from twelve domains. The probability of an exact match is also strongly associated with the response- conditional perplexity of the reference model. Fewer than half of the outputs, in other words, were token for token identical to what the backbone alone would produce.

Two further results frame that divergence. Despite it, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. And repeating the trajectory evaluation with FP32 yields exact trajectory matching on every prompt evaluated. The divergence shows up in the exact text, then, but not in the scores.

The claim, then, is qualified rather than overturned. Under FP32, exact matching holds on all the prompts tested. Under BF16, most trajectories diverge, while benchmark scores show no systematic loss. The study concerns one architecture, examined at these two precisions, on the stated set of prompts and benchmarks. The link to perplexity is reported as an association.

The authors draw a methodological conclusion. The practical losslessness of Orthrus depends on numerical precision, and exact trajectory equivalence should be evaluated separately from downstream task performance, because, as this reproduction shows, one can fail while the other holds.