Deep learning (DL) has had unprecedented success and is now entering scientific computing with full force. However, current DL methods typically suffer from instability, even when universal approximation properties guarantee the existence of stable neural networks (NNs). We address this paradox by demonstrating basic well-conditioned problems in scientific computing where one can prove the existence of NNs with great approximation qualities, however, there does not exist any algorithm, even randomised, that can train (or compute) such a NN. For any positive integers $K > 2$ and $L$, there are cases where simultaneously: (a) no randomised training algorithm can compute a NN correct to $K$ digits with probability greater than $1/2$, (b) there exists a deterministic training algorithm that computes a NN with $K-1$ correct digits, but any such (even randomised) algorithm needs arbitrarily many training data, (c) there exists a deterministic training algorithm that computes a NN with $K-2$ correct digits using no more than $L$ training samples. These results imply a classification theory describing conditions under which (stable) NNs with a given accuracy can be computed by an algorithm. We begin this theory by establishing sufficient conditions for the existence of algorithms that compute stable NNs in inverse problems. We introduce Fast Iterative REstarted NETworks (FIRENETs), which we both prove and numerically verify are stable. Moreover, we prove that only $\mathcal{O}(|\log(\epsilon)|)$ layers are needed for an $\epsilon$-accurate solution to the inverse problem.
翻译:深层学习( DL) 取得了前所未有的成功,现在正在全力进行科学计算。 然而, 目前的 DL 方法通常会不稳定, 即使通用近似特性保证了稳定的神经网络的存在。 我们通过在科学计算中展示基本条件良好的问题来解决这一悖论。 科学计算中,人们可以用高水平的近距离数字来计算NN, 但是, 没有任何算法, 甚至随机化的算法, 可以( 或随机化的) 这样的NN( 或计算) 。 对于任何正整数 $ > 2美元 和 $2 美元, 任何同时出现的情况是:(a) 没有随机化的培训算法能够将NN的校正值转换为$2美元的数字。 这些结果意味着一种确定性的培训算法, 以1K-1美元的正确数字来计算NNNU, 但是任何这种( 随机化的) 算法都需要任意性的培训数据。 (c) 有一种确定性的培训算法, 用美元和正值的正值数字来计算 NNUR 。 这些结果意味着我们只能用一个精确的精确度来计算。