最优化是应用数学的一个分支,主要指在一定条件限制下,选取某种研究方案使目标达到最优的一种方法。最优化问题在当今的军事、工程、管理等领域有着极其广泛的应用。

VIP内容

神经网络在诸多应用领域展现了巨大的潜力,成为当前最热门的研究方向之一。神经网络的训练主要通过求解一个优化问题来完成,但这是一个困难的非线性优化问题,传统的优化理论难以直接应用。在神经网络和优化的交叉领域,长期以来研究人员积累了大量的理论研究知识,不过这些研究或过于理论而不被大部分实践者所了解,或过于偏工程而不被理论学者所理解和欣赏。本文的目的是总结目前对于神经网络优化基本理论和算法的现状,架起理论和实践、优化和机器学习界之间的桥梁。

对苦于调参常感到困惑的工程师而言,本文可以提供一些已有的理论理解以供参考,并提供一些思考的方式。对理论学者而言,本文力图解释其作为数学问题的困难之所在以及目前的理论进展,以期吸引更多研究者投身神经网络优化理论和算法研究。

本文概述了神经网络的算法和优化理论。首先,我们讨论梯度爆炸/消失问题和更一般的谱控制问题,然后讨论实际中常用的解决方案,包括初始化方法和归一化方法。其次,我们回顾用于训练神经网络的一般优化方法,如SGD、自适应梯度方法和大规模分布式训练方法,以及这些算法的现有理论结果。第三,我们回顾了最近关于神经网络训练的全局问题的研究,包括局部极值、模式连接、彩票假设和无限宽度分析等方面的结果。

成为VIP会员查看完整内容
1
43

最新内容

From ancient to modern times, acoustic structures have been used to control the propagation of acoustic waves. However, the design of the acoustic structures has remained widely a time-consuming and computational resource-consuming iterative process. In recent years, Deep Learning has attracted unprecedented attention for its ability to tackle hard problems with huge datasets, which has achieved state-of-the-art results in various tasks. In this work, an acoustic structure design method is proposed based on deep learning. Taking the design of multi-order Helmholtz resonator for instance, we experimentally demonstrate the effectiveness of the proposed method. Our method is not only able to give a very accurate prediction of the geometry of the acoustic structures with multiple strong-coupling parameters, but also capable of improving the performance of evolutionary approaches in optimization for a desired property. Compared with the conventional numerical methods, our method is more efficient, universal and automatic, which has a wide range of potential applications, such as speech enhancement, sound absorption and insulation.

0
0
下载
预览

最新论文

From ancient to modern times, acoustic structures have been used to control the propagation of acoustic waves. However, the design of the acoustic structures has remained widely a time-consuming and computational resource-consuming iterative process. In recent years, Deep Learning has attracted unprecedented attention for its ability to tackle hard problems with huge datasets, which has achieved state-of-the-art results in various tasks. In this work, an acoustic structure design method is proposed based on deep learning. Taking the design of multi-order Helmholtz resonator for instance, we experimentally demonstrate the effectiveness of the proposed method. Our method is not only able to give a very accurate prediction of the geometry of the acoustic structures with multiple strong-coupling parameters, but also capable of improving the performance of evolutionary approaches in optimization for a desired property. Compared with the conventional numerical methods, our method is more efficient, universal and automatic, which has a wide range of potential applications, such as speech enhancement, sound absorption and insulation.

0
0
下载
预览
Top