In this paper, we propose the reproducing activation function to improve deep learning accuracy for various applications ranging from computer vision problems to scientific computing problems. The idea of reproducing activation functions is to employ several basic functions and their learnable linear combination to construct neuron-wise data-driven activation functions for each neuron. Armed with such activation functions, deep neural networks can reproduce traditional approximation tools and, therefore, approximate target functions with a smaller number of parameters than traditional neural networks. In terms of training dynamics of deep learning, reproducing activation functions can generate neural tangent kernels with a better condition number than traditional activation functions lessening the spectral bias of deep learning. As demonstrated by extensive numerical tests, the proposed activation function can facilitate the convergence of deep learning optimization for a solution with higher accuracy than existing deep learning solvers for audio/image/video reconstruction, PDEs, and eigenvalue problems.
翻译:在本文中,我们提出复制激活功能,以提高从计算机视觉问题到科学计算问题等各种应用的深层次学习精确度。复制激活功能的想法是使用几种基本功能及其可学习的线性组合,为每个神经神经元构建神经元数据驱动激活功能。深神经网络在具备这种激活功能的同时,可以复制传统的近似工具,因此,与传统神经网络相比,近似目标功能的参数数量较少。在深层次学习的培训动态方面,复制激活功能可以产生神经核核核内核,比传统激活功能的更好条件,减少深层学习的光谱偏差。如大量数字测试所显示,拟议的激活功能可以促进深度学习优化,以便找到比现有的音频/成像/视频重建、PDEs和egenvaly问题的深层次学习解决方案更加精确的解决方案。