ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

ECHO. Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction. By Yuxuan Zhou and seven others, from Baidu and Independent Researcher. Published on the seventeenth of October. Its code repository is worked out: about two hundred source files, with a readme.

Multi-token prediction trains a language model to look several tokens ahead, not just one. The existing methods, the authors argue, are auxiliary by design. The loss across multiple horizons mostly trains modules bolted onto the backbone, not the backbone itself. And every extra horizon costs another module and another pass.

Efficient Causal Self-Recursion, or ECHO, removes both of those costs. There are no attached modules. The backbone's own final layer predicts every horizon. That means the supervision for future tokens lands on the very parameters that generate the next token. Predicting further ahead stops being a side task attached to the autoregressive objective, and becomes part of it.

Deeper horizons would still get expensive, and that is handled by Dynamic Depth Sampling, or DDS. Rather than training every depth every time, DDS draws the training depth at random, with probabilities that decay as depth increases. Deep horizons are still trained, just less often, which keeps them affordable.

The gains show up on both sides of the objective. In pretraining, adding depth keeps converting into downstream accuracy at the point where auxiliary modules saturate. The improvement concentrates on reasoning and on code, and it comes at half the multi-token training overhead of training every depth.

At inference, the pretrained model serves as its own drafter. It can draft beyond the depth it was trained for. And it keeps its draft-target alignment on open-ended, high-entropy generation, exactly where post-hoc drafters lose theirs. The advantage holds when the token budget is doubled, when pretraining is run a second time, and after supervised finetuning.

One design decision carries everything here. Move the future-token loss off side modules and onto the layer that already makes predictions, and the same weights become both a better pretrained model and a ready-made drafter. The sampling schedule is what keeps that decision cheap enough to use.