VIP内容
神经网络在诸多应用领域展现了巨大的潜力，成为当前最热门的研究方向之一。神经网络的训练主要通过求解一个优化问题来完成，但这是一个困难的非线性优化问题，传统的优化理论难以直接应用。在神经网络和优化的交叉领域，长期以来研究人员积累了大量的理论研究知识，不过这些研究或过于理论而不被大部分实践者所了解，或过于偏工程而不被理论学者所理解和欣赏。本文的目的是总结目前对于神经网络优化基本理论和算法的现状，架起理论和实践、优化和机器学习界之间的桥梁。
对苦于调参常感到困惑的工程师而言，本文可以提供一些已有的理论理解以供参考，并提供一些思考的方式。对理论学者而言，本文力图解释其作为数学问题的困难之所在以及目前的理论进展，以期吸引更多研究者投身神经网络优化理论和算法研究。
本文概述了神经网络的算法和优化理论。首先，我们讨论梯度爆炸/消失问题和更一般的谱控制问题，然后讨论实际中常用的解决方案，包括初始化方法和归一化方法。其次，我们回顾用于训练神经网络的一般优化方法，如SGD、自适应梯度方法和大规模分布式训练方法，以及这些算法的现有理论结果。第三，我们回顾了最近关于神经网络训练的全局问题的研究，包括局部极值、模式连接、彩票假设和无限宽度分析等方面的结果。
成为VIP会员查看完整内容
Optimization for deep learning: an overview
热门内容
最新内容
From ancient to modern times, acoustic structures have been used to control the propagation of acoustic waves. However, the design of the acoustic structures has remained widely a time-consuming and computational resource-consuming iterative process. In recent years, Deep Learning has attracted unprecedented attention for its ability to tackle hard problems with huge datasets, which has achieved state-of-the-art results in various tasks. In this work, an acoustic structure design method is proposed based on deep learning. Taking the design of multi-order Helmholtz resonator for instance, we experimentally demonstrate the effectiveness of the proposed method. Our method is not only able to give a very accurate prediction of the geometry of the acoustic structures with multiple strong-coupling parameters, but also capable of improving the performance of evolutionary approaches in optimization for a desired property. Compared with the conventional numerical methods, our method is more efficient, universal and automatic, which has a wide range of potential applications, such as speech enhancement, sound absorption and insulation.
最新论文
From ancient to modern times, acoustic structures have been used to control the propagation of acoustic waves. However, the design of the acoustic structures has remained widely a time-consuming and computational resource-consuming iterative process. In recent years, Deep Learning has attracted unprecedented attention for its ability to tackle hard problems with huge datasets, which has achieved state-of-the-art results in various tasks. In this work, an acoustic structure design method is proposed based on deep learning. Taking the design of multi-order Helmholtz resonator for instance, we experimentally demonstrate the effectiveness of the proposed method. Our method is not only able to give a very accurate prediction of the geometry of the acoustic structures with multiple strong-coupling parameters, but also capable of improving the performance of evolutionary approaches in optimization for a desired property. Compared with the conventional numerical methods, our method is more efficient, universal and automatic, which has a wide range of potential applications, such as speech enhancement, sound absorption and insulation.
