In this paper, we investigate in the context of deep neural networks, the performance of several discretization algorithms for two first-order finite-time optimization flows. These flows are, namely, the rescaled-gradient flow (RGF) and the signed-gradient flow (SGF), and consist of non-Lipscthiz or discontinuous dynamical systems that converge locally in finite time to the minima of gradient-dominated functions. We introduce three discretization methods for these first-order finite-time flows, and provide convergence guarantees. We then apply the proposed algorithms in training neural networks and empirically test their performances on three standard datasets, namely, CIFAR10, SVHN, and MNIST. Our results show that our schemes demonstrate faster convergences against standard optimization alternatives, while achieving equivalent or better accuracy.
翻译:在本文中,我们从深层神经网络的角度来调查两种一级有限时间优化流的几种离散算法的运作情况,即重新缩放渐渐流(RGF)和签字渐流(SGF),这些流由非利普西斯或不连续的动态系统组成,这些系统在有限的时间内在当地与梯度主导功能的小型相融合。我们对这些一阶定时流采用三种离散方法,并提供汇合保证。然后,我们在培训神经网络时采用拟议的算法,并在三个标准数据集,即CIFAR10、SVHN和MMIST上以经验测试其性能。我们的结果表明,我们的计划比标准优化替代方法更快趋同,同时实现相同或更准确性。