The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

The Life Cycle of a Massive Activation. Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation. By S. Aaron McClendon and two others, from Aimpoint Digital Labs, Mistral AI, and one other institution. Published on the fifth of October. Its code repository is public and real: about thirty source files.

Massive activations are coordinates in a transformer's residual stream whose magnitudes are far larger than typical activations. They are associated with attention sinks, but how their scale is regulated during training has remained incompletely understood. The central claim of this paper is that weight decay causally controls the turnover of that global activation scale: the point at which it stops rising and begins to fall. Around that claim, the authors trace a full life cycle, from emergence, through growth, to consolidation.

The life cycle comes from training-trajectory analyses combined with controlled interventions. Birth is stochastic. Which channels carry the sink varies from one random seed to another, but within any single run those channels stabilise early. Over longer training, the surrounding channels erode and the sink concentrates onto a few redundant carriers. Ablations add a further point: gradient attenuation follows the collective root-mean-square magnitude of the sink token, not any single channel. Collective scale is therefore the quantity that matters for understanding their effects.

The causal evidence comes from controlled continuations of training. Remove weight decay near the peak and the global scale keeps rising. Keep it, and the scale declines, even at a constant learning rate. To explain the rise and the peak, the authors build a balance model in which growth preconditioned by AdamW pushes against weight decay. Sweeping the decay coefficient, lambda, shifts the timing of the peak approximately log-linearly, and the peak magnitude scales approximately as lambda to the minus one half. Both are consistent with the balance. Optimizer measurements show one more thing: preconditioning sustains the cohort of large channels against decay even when the raw maintaining forces would be too small to do so on their own.

The claims are about training dynamics and how activation scale is regulated. The log-linear timing and the inverse square-root scaling are both stated as approximate, and they come from sweeps of the decay coefficient within the authors' controlled settings. The causal result is specifically about weight decay governing when the global scale turns over. Which channels end up carrying the sink is not something the paper claims to predict: that part varies with the seed.

Put together, the picture is of a quantity held in tension rather than one that simply grows. AdamW's preconditioning drives the massive activation up, weight decay pulls it down, and the peak sits where the two balance. That turns weight decay into a training-time lever on activation magnitude, and it ties the whole life cycle, from birth to consolidation, back to the dynamics that regulate scale.