Phase transitions in SGD beyond Lyapunov stability

Deep networks trained with finite step sizes exhibit non-monotonic, oscillatory loss. For full-batch gradient descent this is well charted: the quadratic model puts the stability boundary at twice the inverse step size, and deep neural networks rise to that threshold and hover there.
For SGD, the quadratic model yields not one threshold but several, of which mean-square and Lyapunov stability are the standard candidates, and neither predicts how the distribution of iterates behaves in trained networks. We show across several neural-network experiments that networks can usually train past the Lyapunov threshold, where minibatch-linearized updates amplify deviations. Beyond that point, the nonlinearity of the loss supplies the stabilization and we characterize the stationary phases that emerge because of this.
Among them we identify a new threshold, harmonic instability, set by an inverse-moment condition on the random multipliers. It predicts when the stationary distribution of the iterated phase transits from unimodal to bimodal. An accompanying experiment widget is also provided.