Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models

Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models. By Xin Liu and five others. Published on the third of November.

Some token embeddings in released language models are almost exactly zero, and in gated sequence models that can quietly break training. A near-zero input embedding sets off a chain of gradient amplification, the gradients overflow during clipping, and whole batches of updates are thrown away without any obvious error.

The first piece of evidence is a survey. Across two hundred ninety-one open-weight language model repositories, covering two hundred eighty distinct embedding tables, fifty-two point six percent of untied gated and hybrid checkpoints, twenty out of thirty-eight, contain reachable input embeddings with norms below ten to the minus six. Among weight-tied models, the share is zero. The proposed origin is pretraining: rare or template tokens that never appear get zero input gradients, while AdamW weight decay keeps shrinking the untied embeddings.

Then the mechanism, as the abstract lays it out. In gated sequence models such as Mamba, Gated DeltaNet and xLSTM, a near-zero token can amplify input-embedding gradients by more than thirty orders of magnitude, and up to thirty-five. At zero input, the readout factors for the current token, SiLU gates or linear queries, vanish, which suppresses readout of the historical state. Residual RMS values can then fall below the square root of the RMSNorm floor for several layers in a row, and Pre-RMSNorm amplifies backward gradients by up to a factor of ten to the third at each layer.

The last link is numerical. Once a gradient entry passes one point eight four times ten to the nineteenth, global-norm clipping in FP32 can overflow to infinity, in PyTorch, the Hugging Face Trainer and FSDP2 alike, and the current batch's gradients are zeroed. In Telugu continual pretraining of Nemotron-3-Nano-4B, this throws away updates on about twenty percent of steps and leaves the triggered rows frozen.

The failure is specific to gated sequence models. In the Transformer configurations tested, non-initial positions or a BOS token prevent it. The survey's striking share applies to untied gated and hybrid checkpoints, not to weight-tied models, where none was found. Two remedies are offered, each for a different situation. For checkpoints already released, GroupClip clips the input embeddings separately, using a sixty-four-bit or a rescaled norm. The rescaled version eliminates the lost updates and heals triggered rows in a single step, with negligible measured end-to-end overhead. For pretraining, GateFloor adds a small constant, between zero point zero one and zero point one, to the SiLU gate so that it stays positive at zero input. In experiments up to three hundred seventy million parameters, it reduces near-zero gradients by twenty-eight orders of magnitude, with perplexity comparable to the baseline.

The word that carries this paper is silently. Nothing crashes. A few rows of an embedding table drift to zero during pretraining, a gate closes, a norm amplifies, a clipping routine overflows, and in one continual pretraining run about a fifth of the steps lose their updates. The fix is small once the chain is traced, but the chain only becomes visible by following a single near-zero token all the way through.