We prove two universal approximation theorems for a range of dropout neural networks. These are feed-forward neural networks in which each edge is given a random $\{0,1\}$-valued filter, that have two modes of operation: in the first each edge output is multiplied by its random filter, resulting in a random output, while in the second each edge output is multiplied by the expectation of its filter, leading to a deterministic output. It is common to use the random mode during training and the deterministic mode during testing and prediction. Both theorems are of the following form: Given a function to approximate and a threshold $\varepsilon>0$, there exists a dropout network that is $\varepsilon$-close in probability and in $L^q$. The first theorem applies to dropout networks in the random mode. It assumes little on the activation function, applies to a wide class of networks, and can even be applied to approximation schemes other than neural networks. The core is an algebraic property that shows that deterministic networks can be exactly matched in expectation by random networks. The second theorem makes stronger assumptions and gives a stronger result. Given a function to approximate, it provides existence of a network that approximates in both modes simultaneously. Proof components are a recursive replacement of edges by independent copies, and a special first-layer replacement that couples the resulting larger network to the input. The functions to be approximated are assumed to be elements of general normed spaces, and the approximations are measured in the corresponding norms. The networks are constructed explicitly. Because of the different methods of proof, the two results give independent insight into the approximation properties of random dropout networks. With this, we establish that dropout neural networks broadly satisfy a universal-approximation property.
翻译:我们为一系列辍学神经网络证明了两个通用近距离近似理论。 这些都是向向导神经网络, 向导神经网络, 向向导神经网络提供一种随机的 $0, 1 ⁇ $ $$ 美元, 价值为美元的过滤器, 有两个操作模式: 首先是将每个边缘输出乘以随机过滤器, 导致随机输出, 而第二是将每个边缘输出乘以过滤器的预期, 导致确定性输出。 在培训和预测期间使用随机模式和确定性模式是常见的。 这两种向导都是以下列形式出现的: 以大约和阈值 $\ valepsilon>0 的值计算一个函数, 以随机网络的概率和 $LQQq$为单位, 第一个参数适用于随机模式的退出性能。 它在启动功能上几乎没有多少, 适用于广泛的网络, 并且甚至可以应用到非神经网络的精确性能。 核心是一种向导测算的数学属性, 表明确定确定确定网络的稳定性网络的特性可以与随机网络的预期完全匹配 。 导致一个更精确的网络的精确的精确的网络结果。 第二是一种更强烈的顺序。 将它提供一种更强烈的顺序的功能。 。 。 提供一种更精确的功能。 。 一种更精确的功能 。