Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Percolation Dynamics in Optimization. Variance Cascades and Discrete Scale Invariance. By Sai Niranjan Ramachandran and one other, from the Technical University of Munich. Published on the second of September. No code repository is linked to it.

Stochastic gradient descent is known to steer deep networks toward invariant sets corresponding to simpler subnetworks. That much is established. What remains poorly understood is how the steering unfolds over time, which is a question about the shape of the trajectory rather than about where it ends up.

The paper models the stochastic gradient flow as a percolation process. In that model, architectural symmetries force subnetworks to merge in discrete simultaneous blocks, rather than one at a time as you might expect from a gradual optimization process. And those structural transitions leave a detectable trace: they register as variance spikes in a macroscopic order parameter, echoing physical phase transitions.

Percolation is doing real work there rather than serving as an analogy. It is the standard model for a system where local connections accumulate until connectivity appears suddenly at a threshold, which is the discrete simultaneous merging the symmetries impose here. The argument runs from architectural symmetry to the block structure of the merges, and from that to an observable variance signature, which is the step that makes the account testable.

The trapping mechanism and its scaling cascade extend to Adam and AdamW, but under an explicit heavy-tailed noise model, so that extension is conditional on the noise assumption rather than general. And this is a model of the dynamics: the transitions are said to register as variance spikes, which is a prediction about a signature rather than an observation of subnetworks merging.

The connection drawn is to physics, to phase transitions and discrete scale invariance. If it holds, an apparently smooth optimization is a sequence of sudden structural collapses, and the variance spikes are where to watch them.

Percolation Dynamics in Optimization. Variance Cascades and Discrete Scale Invariance. By Sai Niranjan Ramachandran and one other, from the Technical University of Munich. Published on the second of September. This is the long version: the whole paper rather than the abstract.

Deep networks generalise well without explicit regularization, which is usually attributed to implicit biases introduced during training. Recent work established that stochastic gradient descent is a central source of that bias, collapsing networks onto invariant sets which behave like much simpler subnetworks. What remains poorly understood is the dynamics by which those sets are reached, and this paper argues that gap is not academic. It limits our ability to explain anomalous training behaviour mechanistically.

Grokking is the example they lead with. A transformer memorises the training data perfectly, then only discovers a generalising solution after thousands of further epochs, which challenges the classical picture of smooth optimization. Similar patterns appear in exact deep linear networks and under task shifts. The suggestion drawn from that family of observations is that the topological structure of the dynamics contains crucial information about how optimization actually progresses, and that a smooth-descent picture is missing it.

The construction proceeds in steps that are each individually reasonable. Describe optimization continuously with a stochastic differential equation recording how parameters drift and diffuse over time. Architectural symmetries produce invariant sets which draw trajectories toward them. Independent parameters entering those sets come together and fuse into equivalence classes: the converging weights collide and merge, folding the parameter space into a simpler topology. And then the key representational move. That dynamic is mapped onto a graph tracking merges and splits, a Reeb graph, which is renormalized into a percolation process, where isolated components link and fuse and can cause sudden macroscopic transitions.

Which is exactly what the first contribution does. They determine the precise condition under which the discontinuity survives as the network grows large, rather than being a finite-size artifact that vanishes in the limit. That distinction is the difference between a genuine phenomenon and a curiosity of small models. The framework is developed for stochastic gradient descent, where the authors say the mechanism is cleanest to state and prove, and then extended, because most large-scale training uses adaptive optimizers. The same trapping mechanism and cascade structure are shown to extend to Adam and AdamW under an explicit heavy-tailed noise model, and that extension is what justifies running their grokking experiment with AdamW rather than plain gradient descent.

The formal result is that the process exhibits discrete scale invariance, proved as a theorem, confirmed exactly in toy models, and reported as consistent with the same cascade across three quite different settings: tabular data, vision, and the grokking experiment run with an adaptive optimizer that the earlier extension was needed to justify. That spread matters for a claim about symmetry-induced dynamics, because the mechanism is supposed to come from the architecture rather than from any particular kind of data. The word consistent is doing precise work there and should be heard as weaker than confirmed: the toy models are where the exact confirmation lives, and the realistic settings show the cascade rather than establishing it.

The future-work list is effectively the limitations list, and it is honest about what remains open. Whether these discontinuities persist at practical network widths is named as something still to test, which is a substantial caveat given that surviving the large-width limit was one of the paper's own contributions. The framework covers the invariant manifolds they examine, and extending it to curved manifolds and extreme hyperparameter regimes is future work. The adaptive optimizer extension holds under an explicit heavy-tailed noise assumption rather than generally. And the practical payoffs are proposed rather than demonstrated: using the variance spikes to guide learning rate scheduling, and connecting the cascade to empirical scaling laws as a possible route to deriving new ones.

The picture is that an apparently smooth optimization is a sequence of sudden structural collapses driven by the symmetries of the architecture, with variance spikes marking where they happen. If that holds at practical network width, then grokking stops being an anomaly and becomes one of these transitions that simply arrived late, which would be a genuine explanation rather than a redescription. That conditional is the whole thing, and the paper is admirably clear that it remains a conditional.