Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration

Modern machine learning often requires training with large batch sizes, distributed data, and massively parallel compute hardware. While communication becomes a major bottleneck in such settings, methods like Local Stochastic Gradient Descent (Local SGD) show great promise in reducing this overhead. Local SGD consists of three distinct parts: a local optimization process, an aggregation mechanism, and an outer optimizer that uses aggregated updates to produce a new model.
While extensive literature exists on understanding hyperparameters in the local optimization process, the choice of the outer optimizer and its hyperparameters is less clear. In this talk, we will explore the role of the outer optimizer in Local SGD and present new convergence guarantees for the algorithm. In particular, we will demonstrate that tuning the outer learning rate allows us to (a) trade off between optimization error and stochastic gradient noise variance, and (b) compensate for the ill-tuning of the inner learning rate. Our theory suggests that the outer learning rate should sometimes be set to values greater than 1.
Furthermore, we will extend these results to settings using momentum in the outer optimizer, showing a similar role for the momentum-adjusted outer learning rate. We will also discuss acceleration in the outer optimizer, showing that it improves the convergence rate as a function of the number of communication rounds, improving upon prior algorithms that apply acceleration locally. Finally, we will discuss some experimental validation for our theoretical results and some avenues for future work.