Communications in Information and Systems
Volume 20 (2020)
Mathematical Engineering: A special issue at the occasion of the 85th birthday of Prof. Thomas Kailath
Guest Editors: Ali H. Sayed, Helmut Bölcskei, Patrick Dewilde, Vwani Roychowdhury, and Stephen Shing-Toung Yau
Second-order guarantees in centralized, federated and decentralized nonconvex optimization
Pages: 353 – 388
Rapid advances in data collection and processing capabilities have allowed for the use of increasingly complex models that give rise to nonconvex optimization problems. These formulations, however, can be arbitrarily difficult to solve in general, in the sense that even simply verifying that a given point is a local minimum can be NPhard . Still, some relatively simple algorithms have been shown to lead to surprisingly good empirical results in many contexts of interest. Perhaps the most prominent example is the success of the backpropagation algorithm for training neural networks. Several recent works have pursued rigorous analytical justification for this phenomenon by studying the structure of the nonconvex optimization problems and establishing that simple algorithms, such as gradient descent and its variations, perform well in converging towards local minima and avoiding saddle-points. A key insight in these analyses is that gradient perturbations play a critical role in allowing local descent algorithms to efficiently distinguish desirable from undesirable stationary points and escape from the latter. In this article, we cover recent results on second-order guarantees for stochastic first-order optimization algorithms in centralized, federated, and decentralized architectures.
In honor of Professor Thomas Kailath on the occasion of his 85th birthday.
Received 15 March 2020
Published 2 December 2020