While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$-stationary solution under a constant step size with $\mathcal{O}(1/\epsilon^2)$ computation complexity and $\mathcal{O}(1/\epsilon)$ communication complexity. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal{O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms.
翻译:虽然提出了许多分布式优化算法,以解决网络上的平滑或康纳问题,但其中很少有人能够处理非convex和非movex问题。根据一种近似初始的纯度方法,本文件展示了一种新的(随机)分布式算法,与Nesterov 动力,以加速优化非convex和非湿度问题。理论上,我们显示,拟议的算法可以在一个固定的步数范围内,用美元(1//o}(1/\epsilon2)美元计算复杂性和$\mathcal{O}(1/\epsilon)美元通信复杂性和$\mathcal{O}(1/\epsilon)美元通信复杂性。与现有的基于梯度跟踪方法相比,拟议的算法具有相同的计算复杂性的顺序,但通信复杂性的顺序较低。据我们所知,我们所介绍的结果是,用美元/mathcal{O}(1/\esilonlon)美元(1/\islon)在非conx和非深层的通信问题中可以达到一个持续的通信复杂性。为分布式网络分析问题而提出的神经实验是基于网络的网络分析问题。