Newton-like method with diagonal correction for distributed optimization

Abstract

© 2017 Society for Industrial and Applied Mathematics. We consider distributed optimization problems where networked nodes cooperatively minimize the sum of their locally known convex costs. A popular class of methods to solve these problems are the distributed gradient methods, which are attractive due to their inexpensive itera-tions, but have a drawback of slow convergence rates. This motivates the incorporation of second order information in the distributed methods, but this task is challenging: Although the Hessians which arise in the algorithm design respect the sparsity of the network, their inverses are dense, hence rendering distributed implementations dificult. We overcome this challenge and propose a class of distributed Newton-like methods, which we refer to as distributed quasi-Newton (DQN). The DQN family approximates the Hessian inverse by (1) splitting the Hessian into its diagonal and off-diagonal parts, (2) inverting the diagonal part, and (3) approximating the inverse of the off-diagonal part through a weighted linear function. The approximation is parameterized by the tuning variables which correspond to different splittings of the Hessian and by different weightings of the off-diagonal Hessian part. Specific choices of the tuning variables give rise to different variants of the proposed general DQN method|dubbed DQN-0, DQN-1, and DQN-2|which mutually trade-off communi-cation and computational costs for convergence. Simulations demonstrate the effectiveness of the proposed DQN methods.