Beyond Linearization: On Quadratic and Higher-order Taylor Models of Wide Neural Networks

In this talk, we consider the training of over-parametrized neural networks that are beyond the NTK regime yet still governed by the Taylor expansion of the network. We bring forward the idea of randomizing the neural networks, which allows them to escape their NTK and couple with quadratic models. We show that the optimization landscape of randomized two-layer networks are nice and amenable to escaping-saddle algorithms. We prove concrete generalization and expressivity results on these randomized networks, which lead to sample complexity bounds (of learning certain simple functions) that match the NTK and can in addition be better by a dimension factor when mild distributional assumptions are present. We present additional experimental results on the approximation power of higher-order Taylor models, and theoretical results showing that when we increase the depth, quadratic models can further benefit in the sample complexity of learning certain low-rank polynomials.