We propose a new unified framework for describing and designing gradient-based convex optimization methods from a numerical analysis perspective. There the key is the new concept of weak discrete gradients (weak DGs), which is a generalization of DGs standard in numerical analysis. Via weak DG, we consider abstract optimization methods, and prove unified convergence rate estimates that hold independent of the choice of weak DGs except for some constants in the final estimate. With some choices of weak DGs, we can reproduce many popular existing methods, such as the steepest descent and Nesterov's accelerated gradient method, and also some recent variants from numerical analysis community. By considering new weak DGs, we can easily explore new theoretically-guaranteed optimization methods; we show some examples. We believe this work is the first attempt to fully integrate research branches in optimization and numerical analysis areas, so far independently developed.
翻译:我们从数字分析的角度提出了一个新的描述和设计基于梯度的曲线优化方法的统一框架。关键在于弱离性梯度的新概念(较弱的DGs),这是数字分析中DGs标准的一般化。通过弱DG,我们考虑抽象的优化方法,并证明统一的趋同率估计数,这些估计数与选择较弱的DG不相干,但最后估计中的一些常数除外。由于对较弱的DG作了一些选择,我们可以复制许多流行的现有方法,例如最陡峭的下降和Nesterov的加速梯度方法,以及数字分析界最近的一些变异。通过考虑新的弱的DGs,我们可以很容易地探索新的理论上有保障的优化方法;我们举一些例子。我们认为,这项工作是将研究分支充分纳入优化和数字分析领域,迄今独立发展起来的第一次尝试。